Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Critique our use of curl; and HTTP/2, /3, multiplexing & pipelining questions
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Stefan Eissing via curl-library <curl-library_at_lists.haxx.se>
Date: Wed, 22 Feb 2023 19:23:42 +0100
> Am 22.02.2023 um 17:01 schrieb Daniel Stenberg via curl-library <curl-library_at_lists.haxx.se>:
>
> On Tue, 21 Feb 2023, Richard W.M. Jones via curl-library wrote:
>
>> Firstly, I don't understand if the multi interface would actually help us here. Because nbdkit gives us lots of threads and expects an NBD request to be processed synchronously on that thread, using the easy interface is a natural .. easy(!) .. fit.
>
> Doing multiple transfers using the multi interface instead of separate threads using the easy interface has some pros and some cons.
>
> Pros:
>
> - shared pools and caches
> - you can multiplex transfers with h2/h3
>
> Cons:
>
> - multiple threads running on multiple cores can perform faster in case you
> are CPU bound
There is no one size fits all here, as Daniel indicates. If you register callbacks in curl to get your answers and expect them to be called on a particular thread, you need to run curl in that thread.
The transfer sizes matter as well. If you have many small transfers, multiplexing can really help. For large transfers, the gains are less. But then again, if you, have a small request in the "middle" it will not need to wait on the large ones to complete.
As you can see, it really depends on your application, how it processes the responses, what kind of requests it makes and what response times it needs.
>> A second thing I'm unclear about with multi is whether the individual easy handles which are added are related in any way -- eg. if they all share the same TCP connection to the web server?
>
> The *can* share the same conncetion with HTTP/2 and HTTP/3 multiplexing, yes.
>
>> Reading the page makes me think this is not the case, the multi interface is just a way to group easy handles for the purposes of using a select/poll or event-driven API, and apart from that there is no relationship.
>
> They also share connection pool, DNS cache, CA cert cache etc.
>
>> The third and main concern is whether we are using curl most efficiently. In particular, whether we are using HTTP/2 (and in future HTTP/3) as efficiently as we could be (eg. exploiting multiplexing).
>
> For multiplexing you *must* use the multi interface and add multiple transfers to it.
>
>> I notice that HTTP/1.1-style pipelining was removed from curl, and I suppose HTTP/2 multiplexing is meant to replace this. However since we are using the easy interface and doing everything synchronously, it's my understanding that we are not exploiting multiplexing, unless curl itself does something clever internally.
>
> It can't do anything clever when for synchronous serial transfers. Then you can't do multiplexing.
>
> --
>
> / daniel.haxx.se
> | Commercial curl support up to 24x7 is available!
> | Private help, bug fixes, support, ports, new features
> | https://curl.se/support.html
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
Date: Wed, 22 Feb 2023 19:23:42 +0100
> Am 22.02.2023 um 17:01 schrieb Daniel Stenberg via curl-library <curl-library_at_lists.haxx.se>:
>
> On Tue, 21 Feb 2023, Richard W.M. Jones via curl-library wrote:
>
>> Firstly, I don't understand if the multi interface would actually help us here. Because nbdkit gives us lots of threads and expects an NBD request to be processed synchronously on that thread, using the easy interface is a natural .. easy(!) .. fit.
>
> Doing multiple transfers using the multi interface instead of separate threads using the easy interface has some pros and some cons.
>
> Pros:
>
> - shared pools and caches
> - you can multiplex transfers with h2/h3
>
> Cons:
>
> - multiple threads running on multiple cores can perform faster in case you
> are CPU bound
There is no one size fits all here, as Daniel indicates. If you register callbacks in curl to get your answers and expect them to be called on a particular thread, you need to run curl in that thread.
The transfer sizes matter as well. If you have many small transfers, multiplexing can really help. For large transfers, the gains are less. But then again, if you, have a small request in the "middle" it will not need to wait on the large ones to complete.
As you can see, it really depends on your application, how it processes the responses, what kind of requests it makes and what response times it needs.
>> A second thing I'm unclear about with multi is whether the individual easy handles which are added are related in any way -- eg. if they all share the same TCP connection to the web server?
>
> The *can* share the same conncetion with HTTP/2 and HTTP/3 multiplexing, yes.
>
>> Reading the page makes me think this is not the case, the multi interface is just a way to group easy handles for the purposes of using a select/poll or event-driven API, and apart from that there is no relationship.
>
> They also share connection pool, DNS cache, CA cert cache etc.
>
>> The third and main concern is whether we are using curl most efficiently. In particular, whether we are using HTTP/2 (and in future HTTP/3) as efficiently as we could be (eg. exploiting multiplexing).
>
> For multiplexing you *must* use the multi interface and add multiple transfers to it.
>
>> I notice that HTTP/1.1-style pipelining was removed from curl, and I suppose HTTP/2 multiplexing is meant to replace this. However since we are using the easy interface and doing everything synchronously, it's my understanding that we are not exploiting multiplexing, unless curl itself does something clever internally.
>
> It can't do anything clever when for synchronous serial transfers. Then you can't do multiplexing.
>
> --
>
> / daniel.haxx.se
> | Commercial curl support up to 24x7 is available!
> | Private help, bug fixes, support, ports, new features
> | https://curl.se/support.html
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2023-02-22