Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
Re: How can libcurl/curl_multi perform like curl --parallel ?
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Oliver Schonrock via curl-library <curl-library_at_lists.haxx.se>
Date: Fri, 25 Oct 2024 16:14:39 +0100
On 25/10/2024 08:58, Oliver Schonrock via curl-library wrote:
>
> What is `curl --parallel --parallel-max 150` doing internally and how
> can I reproduce this performance with libcurl?
Answering my own question...
Some investigating with `perf` showed that the 100% CPU process was
spending almost all its time negotiating encrypted connections.
Enabling CURLOPT_VERBOSE showed that it was opening a separate encrypted
connected for each download.
Comparing that to the `curl --parallel --verbose` case, showed that
`curl` only negotiated a single connection and then did HTTP2
multiplexed streaming just as I had suspected.
Some experimentation showed that setting
CURLMOPT_MAX_TOTAL_CONNECTIONS = 1
thereby limiting curl_multi to a single encrypted connection, and
forcing libcurl to multiplex streams over HTTP2.
So now libcurl delivers very similar performance to `curl --parallel`.
Some additional tuning of
CURLMOPT_MAX_CONCURRENT_STREAMS
will probably be required, perhaps to match the --parallel-max=150`, to
optimise the full sized download.
All good.
Date: Fri, 25 Oct 2024 16:14:39 +0100
On 25/10/2024 08:58, Oliver Schonrock via curl-library wrote:
>
> What is `curl --parallel --parallel-max 150` doing internally and how
> can I reproduce this performance with libcurl?
Answering my own question...
Some investigating with `perf` showed that the 100% CPU process was
spending almost all its time negotiating encrypted connections.
Enabling CURLOPT_VERBOSE showed that it was opening a separate encrypted
connected for each download.
Comparing that to the `curl --parallel --verbose` case, showed that
`curl` only negotiated a single connection and then did HTTP2
multiplexed streaming just as I had suspected.
Some experimentation showed that setting
CURLMOPT_MAX_TOTAL_CONNECTIONS = 1
thereby limiting curl_multi to a single encrypted connection, and
forcing libcurl to multiplex streams over HTTP2.
So now libcurl delivers very similar performance to `curl --parallel`.
Some additional tuning of
CURLMOPT_MAX_CONCURRENT_STREAMS
will probably be required, perhaps to match the --parallel-max=150`, to
optimise the full sized download.
All good.
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2024-10-25