cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: curl-library mailing list

From: Azat Khuzhin <dohardgopro_at_gmail.com>
Date: Sun, 24 Feb 2013 03:47:36 +0400

Sorry missed message footer.
So you have linux.

What you can check:
- open files limit:
  ulimit -n 32768
  See also /etc/security/limits.conf

- Port range:
  sysctl net.ipv4.ip_local_port_range='1025 65535'

- And also turn off conntrack kernel module if it is loaded. (just
google how to disable it, if you would have trouble with this fill
free to ask)

And after you done all of this, is problem still reproduced?

And also there is some number of parameters that you must to change
for create/maintain many sockets, but I think that information above
must be enough for you.
Because I don't know what do you mean by "many" in you mail.

On Sun, Feb 24, 2013 at 3:41 AM, Azat Khuzhin <dohardgopro_at_gmail.com> wrote:
> Hi Yakov.
>
> What operation system do you have?
>
> Also what can be wrong regardless OS resolver can ban you because of
> too high rate.
> For example google dns did this.
>
> On Sun, Feb 24, 2013 at 1:13 AM, Яков <jack__at_inbox.ru> wrote:
>> Hi everybody.
>> I use libcurl to get very many html-pages via multi interface. And the
>> problem is that when I add to CURLM* handle a lot of easy handles (for
>> example, some thousands), or create many threads to divide it and accelerate
>> my program, for most of them (about 70-80 %) libcurl returns error "Couldn't
>> resolve host name".
>>
>> But when I select some URLs from the list with this error (for example 10-15
>> URLs) and check their via browser or curl (command line) or libcurl via my
>> program - all of them are available and no errors occurs! I can load pages
>> successfully!
>>
>> My code like this approximately:
>> CURLM *mc;
>> //initialize multi handle,
>> //initialize easy handles with `curl_easy_setopt`
>>
>> //adding it to multi interface with `curl_multi_add_handle`
>>
>> CURLMsg *msg = 0;
>> int in_process = -1;
>> int msg_queue = 0, nums = 0;
>>
>> while ( in_process )
>> {
>> curl_multi_perform(mc, &in_process);
>>
>> if ( in_process )
>> {
>> int nums;
>> curl_multi_wait(mc, 0, 0, 500, &nums);
>> }
>>
>> while ( (msg = curl_multi_info_read(mc, &msg_queue)) )
>> {
>> //process message
>> }
>> }
>>
>> //cleanup
>>
>> libcurl 7.28.1, CentOS 5.9 x86_64 and CentOS 6.0 i686.
>>
>> Please, can anybody help me with this problem? Thanks in advance.
>>
>>
>> -------------------------------------------------------------------
>> List admin: http://cool.haxx.se/list/listinfo/curl-library
>> Etiquette: http://curl.haxx.se/mail/etiquette.html
>
>
>
> --
> Respectfully
> Azat Khuzhin
> Primary email a3at.mail_at_gmail.com

--
Respectfully
Azat Khuzhin
Primary email a3at.mail_at_gmail.com
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Received on 2013-02-24