curl / Mailing Lists / curl-library / Single Mail
Buy commercial curl support from WolfSSL. We help you work out your issues, debug your libcurl applications, use the API, port to new platforms, add new features and more. With a team lead by the curl founder himself.

Re: Critique our use of curl; and HTTP/2, /3, multiplexing & pipelining questions

From: Richard W.M. Jones via curl-library <curl-library_at_lists.haxx.se>
Date: Thu, 23 Feb 2023 18:12:17 +0000

On Wed, Feb 22, 2023 at 07:23:42PM +0100, Stefan Eissing via curl-library wrote:
> There is no one size fits all here, as Daniel indicates. If you
> register callbacks in curl to get your answers and expect them to be
> called on a particular thread, you need to run curl in that thread.
>
> The transfer sizes matter as well. If you have many small transfers,
> multiplexing can really help. For large transfers, the gains are
> less. But then again, if you, have a small request in the "middle"
> it will not need to wait on the large ones to complete.
>
> As you can see, it really depends on your application, how it
> processes the responses, what kind of requests it makes and what
> response times it needs.

Thanks, and yes this is a very interesting point. I only implied it
in my original question, but we are using small HTTP range requests
(CURLOPT_RANGE).

The request size depends, but typically you'd see requests ranging
from the qcow2 cluster size (32K) up to the nbdcopy default request
size (256K). In theory it could be [512b..32M].

The idea behind this is that we can boot a machine off a web server,
downloading only the sectors/clusters as needed (with a lot of caching
going on).

In more advanced cases we're able to inspect a remote disk image,
determine which parts are unused (by using filesystem metadata within
the image), and then download only the parts which contain data, even
modifying the disk image filesystem content in flight, and do all of
this while streaming and making no local copies. It's quite an
interesting research area.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html
Received on 2023-02-23