Re: Critique our use of curl; and HTTP/2, /3, multiplexing & pipelining questions

From: Richard W.M. Jones via curl-library <curl-library_at_lists.haxx.se>
Date: Thu, 23 Feb 2023 17:59:53 +0000

On Wed, Feb 22, 2023 at 05:01:00PM +0100, Daniel Stenberg wrote:
> On Tue, 21 Feb 2023, Richard W.M. Jones via curl-library wrote:
>
> >Firstly, I don't understand if the multi interface would actually
> >help us here. Because nbdkit gives us lots of threads and expects
> >an NBD request to be processed synchronously on that thread, using
> >the easy interface is a natural .. easy(!) .. fit.
>
> Doing multiple transfers using the multi interface instead of
> separate threads using the easy interface has some pros and some
> cons.
>
> Pros:
>
> - shared pools and caches
> - you can multiplex transfers with h2/h3
>
> Cons:
>
> - multiple threads running on multiple cores can perform faster in case you
> are CPU bound
>
> >A second thing I'm unclear about with multi is whether the
> >individual easy handles which are added are related in any way --
> >eg. if they all share the same TCP connection to the web server?
>
> The *can* share the same conncetion with HTTP/2 and HTTP/3 multiplexing, yes.

Does this sharing happen without us doing anything, or do we need to
use the curl share interface? I only discovered share after sending
the original message yesterday. I then patched nbdkit to use it:

https://listman.redhat.com/archives/libguestfs/2023-February/030839.html

but it doesn't seem to make any noticable difference in my performance
tests.

At the same time I thought I'd actually set up an HTTP/2 webserver
(just a local Apache 2.4) so I could see how curl works against these.
But I get very strange problems (with or without the above patch)
where the server appears to report 403 errors at random, and curl
itself can deadlock. I'm sorry for not being very specific on this as
I'm still investigating, but should we expect curl + HTTP/2 to work
well, or is it still experimental? If it's expected to work then I'll
try to investigate further, use upstream curl, and see if I can make a
proper reproducer.

> >Reading the page makes me think this is not the case, the multi
> >interface is just a way to group easy handles for the purposes of
> >using a select/poll or event-driven API, and apart from that there
> >is no relationship.
>
> They also share connection pool, DNS cache, CA cert cache etc.
>
> >The third and main concern is whether we are using curl most
> >efficiently. In particular, whether we are using HTTP/2 (and in
> >future HTTP/3) as efficiently as we could be (eg. exploiting
> >multiplexing).
>
> For multiplexing you *must* use the multi interface and add multiple
> transfers to it.

The share interface (instead of multi) _won't_ give us HTTP/2
multiplexing?

> >I notice that HTTP/1.1-style pipelining was removed from curl, and
> >I suppose HTTP/2 multiplexing is meant to replace this. However
> >since we are using the easy interface and doing everything
> >synchronously, it's my understanding that we are not exploiting
> >multiplexing, unless curl itself does something clever internally.
>
> It can't do anything clever when for synchronous serial transfers.
> Then you can't do multiplexing.

If we need to use the multi interface then I can try coding that up
later.

Thanks,

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2023-02-23