Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: curl-multi-socket interface too slow
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Patrick Schlangen via curl-library <curl-library_at_lists.haxx.se>
Date: Wed, 1 Jun 2022 18:59:53 +0000
Am 01.06.2022 um 20:42 schrieb Henrik Holst via curl-library <curl-library_at_lists.haxx.se>:
>
> I wonder if you in that specific example (the stackexchange one) are hitting some limit in the dns resolver since "domains/s=161.98" which is the max value there would be something like (1 event for dns reply, 1 for connect, 1 for write is what I'm guesstimating at here) ~486 events in epoll per second is not much events at all.
>
> I wouldn't be at all surprised if public DNS servers such as the Cloudflare and Google one used in your example have some kind of rate limiting.
I'm running a similar usage pattern of accessing random domain names with epoll (behind libev) based I/O and, according to ifstat, see around 82 Mbit/s peak incoming and 20 Mbit/s outgoing bandwidth when using 2 threads with 1024 parallel requests each. I'm using c-ares as a resolver and unbound as a DNS cache. In addition, there's some sysctl tuning involved:
net.core.somaxconn = 256
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.ip_local_port_range = 1024 65535
net.core.netdev_max_backlog = 2500
net.core.rmem_max = 26214400
net.core.rmem_default = 26214400
net.core.wmem_max = 26214400
This [1] is running on a pretty weak virtual server. Especially the rmem/wmem max settings were crucial for being able to cope with the UDP traffic between curl and unbound. (Otherwise UDP packets seem to get lost before they are processed and this leads to DNS timeouts.)
Just posting this here because it seems I'm getting much better performance than what was mentioned in the stackoverflow thread (which mentioned around 5-6 Mbit/s).
Maybe this is helpful.
Best,
Patrick
[1] https://github.com/pschlan/cron-job.org/blob/master/chronos/CurlWorker.cpp + https://github.com/pschlan/cron-job.org/blob/master/chronos/HTTPRequest.cpp
Date: Wed, 1 Jun 2022 18:59:53 +0000
Am 01.06.2022 um 20:42 schrieb Henrik Holst via curl-library <curl-library_at_lists.haxx.se>:
>
> I wonder if you in that specific example (the stackexchange one) are hitting some limit in the dns resolver since "domains/s=161.98" which is the max value there would be something like (1 event for dns reply, 1 for connect, 1 for write is what I'm guesstimating at here) ~486 events in epoll per second is not much events at all.
>
> I wouldn't be at all surprised if public DNS servers such as the Cloudflare and Google one used in your example have some kind of rate limiting.
I'm running a similar usage pattern of accessing random domain names with epoll (behind libev) based I/O and, according to ifstat, see around 82 Mbit/s peak incoming and 20 Mbit/s outgoing bandwidth when using 2 threads with 1024 parallel requests each. I'm using c-ares as a resolver and unbound as a DNS cache. In addition, there's some sysctl tuning involved:
net.core.somaxconn = 256
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.ip_local_port_range = 1024 65535
net.core.netdev_max_backlog = 2500
net.core.rmem_max = 26214400
net.core.rmem_default = 26214400
net.core.wmem_max = 26214400
This [1] is running on a pretty weak virtual server. Especially the rmem/wmem max settings were crucial for being able to cope with the UDP traffic between curl and unbound. (Otherwise UDP packets seem to get lost before they are processed and this leads to DNS timeouts.)
Just posting this here because it seems I'm getting much better performance than what was mentioned in the stackoverflow thread (which mentioned around 5-6 Mbit/s).
Maybe this is helpful.
Best,
Patrick
[1] https://github.com/pschlan/cron-job.org/blob/master/chronos/CurlWorker.cpp + https://github.com/pschlan/cron-job.org/blob/master/chronos/HTTPRequest.cpp
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.htmlReceived on 2022-06-01