Re: curl-multi-socket interface too slow

From: Henrik Holst via curl-library <curl-library_at_lists.haxx.se>
Date: Wed, 1 Jun 2022 20:42:01 +0200

I wonder if you in that specific example (the stackexchange one) are
hitting some limit in the dns resolver since "domains/s=161.98" which is
the max value there would be something like (1 event for dns reply, 1 for
connect, 1 for write is what I'm guesstimating at here) ~486 events in
epoll per second is not much events at all.

I wouldn't be at all surprised if public DNS servers such as the Cloudflare
and Google one used in your example have some kind of rate limiting.

/HH

Den ons 1 juni 2022 kl 19:42 skrev James Read via curl-library <
curl-library_at_lists.haxx.se>:

>
>
> On Wed, Jun 1, 2022 at 2:22 PM Aleh Linkin <aleh.linkin_at_idt.net> wrote:
>
>> Thanks for the update. Could you be a bit more specific? What do you mean
>> under "problem with the TCP/IP stack"? If I'd like to use
>> curl_multi_socket_action I had to sacrifice some CPU?
>>
>
> Some time ago I posted a question related on stackoverflow:
> https://stackoverflow.com/questions/70584121/why-doesnt-my-epoll-based-program-improve-performance-by-increasing-the-number
>
>
> The question includes no cURL based code which demonstrates the
> limitations of epoll with the current TCP/IP stack. I don't know exactly
> what the problem with the stack is. I'm still working on it.
>
> James Read
>
>
>>
>> On Wed, Jun 1, 2022 at 4:04 PM James Read <jamesread5737_at_gmail.com>
>> wrote:
>>
>>> The problem is not with cURL. I have also run into these problems using
>>> cURL. I recoded using sockets and epoll and got the same results. The
>>> problem is with the TCP/IP stack.
>>>
>>> On Wed, Jun 1, 2022 at 1:46 PM Aleh Linkin via curl-library <
>>> curl-library_at_lists.haxx.se> wrote:
>>>
>>>> 1. easy_handles
>>>> ```
>>>> SET_OPT(CURLOPT_ERRORBUFFER, m_errorBuffer);
>>>> setVerbose(true);
>>>> setUrl(url);
>>>> setTimeout(timeout);
>>>> setConnectTimeout(connectTimeout);
>>>> setProxy(proxy);
>>>> setProxyPort(proxyPort);
>>>> setHttpHeaders(httpHeaders);
>>>> SET_OPT(CURLOPT_WRITEFUNCTION, writeFunction);
>>>> SET_OPT(CURLOPT_WRITEDATA, m_requestState);
>>>> SET_OPT(CURLOPT_DEBUGFUNCTION, CurlRequest::debugFunction);
>>>> SET_OPT(CURLOPT_DEBUGDATA, this);
>>>> SET_OPT(CURLOPT_PRIVATE, this);
>>>> SET_OPT(CURLOPT_NOSIGNAL, 1L);
>>>> ```
>>>>
>>>> 2. HTTPS. TLS based on NSS SSL backend
>>>>
>>>> 3. Yes. I've used perf and valgrind-cachgrind but I can't take the
>>>> reason from results. For example perf part:
>>>> ```
>>>> Samples: 1K of event 'cycles:uppp', Event count (approx.): 9397031894
>>>> Children Self Command Shared Object Symbol
>>>> - 23.25% 2.43% lecappqry.1 [unknown] [.]
>>>> 0000000000000000
>>>> - 21.58% 0
>>>> 5.29% _int_malloc
>>>> 1.42% free
>>>> 1.13% Curl_http
>>>> 1.07% _int_free
>>>> + 0.88% __clock_gettime
>>>> 0.72% _IO_vsnprintf
>>>> 0.71% __libc_calloc
>>>> + 5.07% 0.00% lecappqry.1 [unknown] [k]
>>>> 0x0000000000667849
>>>> + 5.07% 0.00% lecappqry.1 [unknown] [k]
>>>> 0x00007f33507ad900
>>>> + 5.07% 4.87% lecappqry.1 lecappqry_grpc [.]
>>>> dprintf_formatf
>>>> + 4.88% 4.71% lecappqry.1 libc-2.17.so [.] _int_free
>>>> ```
>>>>
>>>> On Mon, May 30, 2022 at 9:07 PM Patrick Schlangen <patrick_at_schlangen.me>
>>>> wrote:
>>>>
>>>>> Am 30.05.2022 um 16:23 schrieb Aleh Linkin via curl-library <
>>>>> curl-library_at_lists.haxx.se>:
>>>>> >
>>>>> > I understand. Thank you.
>>>>> > But why multi_socket_action eats about twice time more CPU then
>>>>> curl_multi_perform called in cycle with 10ms pause?
>>>>>
>>>>> Can you supply some more details about how the easy handle is
>>>>> configured and the type of transfers you are running? Is it HTTP? Are you
>>>>> using TLS? Did you use perf or a similar profiler yet to see where time is
>>>>> spent?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Patrick
>>>>>
>>>>> --
>>>> Unsubscribe: https://lists.haxx.se/listinfo/curl-library
>>>> Etiquette: https://curl.haxx.se/mail/etiquette.html
>>>>
>>> --
> Unsubscribe: https://lists.haxx.se/listinfo/curl-library
> Etiquette: https://curl.haxx.se/mail/etiquette.html
>

-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Received on 2022-06-01