Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Total http/2 concurrency for multiplexed multi-handle
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Daniel Stenberg via curl-library <curl-library_at_lists.haxx.se>
Date: Fri, 10 Feb 2023 18:39:55 +0100 (CET)
On Fri, 10 Feb 2023, Jeroen Ooms wrote:
> Ah ok that is better than I thought. I was under the impression that it
> would immediately start with 6 connections, even before considering
> multiplexing.
There is also CURLOPT_PIPEWAIT that you can use to make libcurl prefer
multiplexing to starting new connections.
> However I noticed that even when setting CURLMOPT_MAX_HOST_CONNECTIONS to 1,
> GitHub still drops the connections at some point. Perhaps the issue isn't
> just the concurrency but we are hitting a proxy_read_timeout or something,
> because the downloads are idling for too long, due to the high
> concurrency...
On a single connection libcurl should fill it up to the max amount of streams,
and none of those should go idle. The transfers that haven't been able to
start yet because of the traffic jam are just qeueud up in memory by libcurl
and will not appear as idle to anyone as they haven't actually started.
> I did find that the problems disappear when I disable multiplexing, and
> performance isn't much worse (about 6 minutes for downloading the 25k
> files), so this solves my immediate problem.
If you're not terribly far away and each transfer mostly saturates the pipe,
doing them serially is not going to be much different than parallel in the
grand total. Multiplexing should still be slightly faster since you won't get
punished by the RTT gaps and slow-starts that serial transfers get, but if
that's just small parts of a second, it might not add up to much.
Date: Fri, 10 Feb 2023 18:39:55 +0100 (CET)
On Fri, 10 Feb 2023, Jeroen Ooms wrote:
> Ah ok that is better than I thought. I was under the impression that it
> would immediately start with 6 connections, even before considering
> multiplexing.
There is also CURLOPT_PIPEWAIT that you can use to make libcurl prefer
multiplexing to starting new connections.
> However I noticed that even when setting CURLMOPT_MAX_HOST_CONNECTIONS to 1,
> GitHub still drops the connections at some point. Perhaps the issue isn't
> just the concurrency but we are hitting a proxy_read_timeout or something,
> because the downloads are idling for too long, due to the high
> concurrency...
On a single connection libcurl should fill it up to the max amount of streams,
and none of those should go idle. The transfers that haven't been able to
start yet because of the traffic jam are just qeueud up in memory by libcurl
and will not appear as idle to anyone as they haven't actually started.
> I did find that the problems disappear when I disable multiplexing, and
> performance isn't much worse (about 6 minutes for downloading the 25k
> files), so this solves my immediate problem.
If you're not terribly far away and each transfer mostly saturates the pipe,
doing them serially is not going to be much different than parallel in the
grand total. Multiplexing should still be slightly faster since you won't get
punished by the RTT gaps and slow-starts that serial transfers get, but if
that's just small parts of a second, it might not add up to much.
-- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2023-02-10