curl / Mailing Lists / curl-library / Single Mail
Buy commercial curl support from WolfSSL. We help you work out your issues, debug your libcurl applications, use the API, port to new platforms, add new features and more. With a team lead by the curl founder himself.

Re: curl-multi-socket interface too slow

From: Henrik Holst via curl-library <curl-library_at_lists.haxx.se>
Date: Wed, 1 Jun 2022 20:42:01 +0200

I wonder if you in that specific example (the stackexchange one) are
hitting some limit in the dns resolver since "domains/s=161.98" which is
the max value there would be something like (1 event for dns reply, 1 for
connect, 1 for write is what I'm guesstimating at here) ~486 events in
epoll per second is not much events at all.

I wouldn't be at all surprised if public DNS servers such as the Cloudflare
and Google one used in your example have some kind of rate limiting.

/HH

Den ons 1 juni 2022 kl 19:42 skrev James Read via curl-library <
curl-library_at_lists.haxx.se>:

>
>
> On Wed, Jun 1, 2022 at 2:22 PM Aleh Linkin <aleh.linkin_at_idt.net> wrote:
>
>> Thanks for the update. Could you be a bit more specific? What do you mean
>> under "problem with the TCP/IP stack"? If I'd like to use
>> curl_multi_socket_action I had to sacrifice some CPU?
>>
>
> Some time ago I posted a question related on stackoverflow:
> https://stackoverflow.com/questions/70584121/why-doesnt-my-epoll-based-program-improve-performance-by-increasing-the-number
>
>
> The question includes no cURL based code which demonstrates the
> limitations of epoll with the current TCP/IP stack. I don't know exactly
> what the problem with the stack is. I'm still working on it.
>
> James Read
>
>
>>
>> On Wed, Jun 1, 2022 at 4:04 PM James Read <jamesread5737_at_gmail.com>
>> wrote:
>>
>>> The problem is not with cURL. I have also run into these problems using
>>> cURL. I recoded using sockets and epoll and got the same results. The
>>> problem is with the TCP/IP stack.
>>>
>>> On Wed, Jun 1, 2022 at 1:46 PM Aleh Linkin via curl-library <
>>> curl-library_at_lists.haxx.se> wrote:
>>>
>>>> 1. easy_handles
>>>> ```
>>>> SET_OPT(CURLOPT_ERRORBUFFER, m_errorBuffer);
>>>> setVerbose(true);
>>>> setUrl(url);
>>>> setTimeout(timeout);
>>>> setConnectTimeout(connectTimeout);
>>>> setProxy(proxy);
>>>> setProxyPort(proxyPort);
>>>> setHttpHeaders(httpHeaders);
>>>> SET_OPT(CURLOPT_WRITEFUNCTION, writeFunction);
>>>> SET_OPT(CURLOPT_WRITEDATA, m_requestState);
>>>> SET_OPT(CURLOPT_DEBUGFUNCTION, CurlRequest::debugFunction);
>>>> SET_OPT(CURLOPT_DEBUGDATA, this);
>>>> SET_OPT(CURLOPT_PRIVATE, this);
>>>> SET_OPT(CURLOPT_NOSIGNAL, 1L);
>>>> ```
>>>>
>>>> 2. HTTPS. TLS based on NSS SSL backend
>>>>
>>>> 3. Yes. I've used perf and valgrind-cachgrind but I can't take the
>>>> reason from results. For example perf part:
>>>> ```
>>>> Samples: 1K of event 'cycles:uppp', Event count (approx.): 9397031894
>>>> Children Self Command Shared Object Symbol
>>>> - 23.25% 2.43% lecappqry.1 [unknown] [.]
>>>> 0000000000000000
>>>> - 21.58% 0
>>>> 5.29% _int_malloc
>>>> 1.42% free
>>>> 1.13% Curl_http
>>>> 1.07% _int_free
>>>> + 0.88% __clock_gettime
>>>> 0.72% _IO_vsnprintf
>>>> 0.71% __libc_calloc
>>>> + 5.07% 0.00% lecappqry.1 [unknown] [k]
>>>> 0x0000000000667849
>>>> + 5.07% 0.00% lecappqry.1 [unknown] [k]
>>>> 0x00007f33507ad900
>>>> + 5.07% 4.87% lecappqry.1 lecappqry_grpc [.]
>>>> dprintf_formatf
>>>> + 4.88% 4.71% lecappqry.1 libc-2.17.so [.] _int_free
>>>> ```
>>>>
>>>> On Mon, May 30, 2022 at 9:07 PM Patrick Schlangen <patrick_at_schlangen.me>
>>>> wrote:
>>>>
>>>>> Am 30.05.2022 um 16:23 schrieb Aleh Linkin via curl-library <
>>>>> curl-library_at_lists.haxx.se>:
>>>>> >
>>>>> > I understand. Thank you.
>>>>> > But why multi_socket_action eats about twice time more CPU then
>>>>> curl_multi_perform called in cycle with 10ms pause?
>>>>>
>>>>> Can you supply some more details about how the easy handle is
>>>>> configured and the type of transfers you are running? Is it HTTP? Are you
>>>>> using TLS? Did you use perf or a similar profiler yet to see where time is
>>>>> spent?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Patrick
>>>>>
>>>>> --
>>>> Unsubscribe: https://lists.haxx.se/listinfo/curl-library
>>>> Etiquette: https://curl.haxx.se/mail/etiquette.html
>>>>
>>> --
> Unsubscribe: https://lists.haxx.se/listinfo/curl-library
> Etiquette: https://curl.haxx.se/mail/etiquette.html
>


-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html
Received on 2022-06-01