Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Critique our use of curl; and HTTP/2, /3, multiplexing & pipelining questions
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Richard W.M. Jones via curl-library <curl-library_at_lists.haxx.se>
Date: Thu, 23 Feb 2023 18:12:17 +0000
On Wed, Feb 22, 2023 at 07:23:42PM +0100, Stefan Eissing via curl-library wrote:
> There is no one size fits all here, as Daniel indicates. If you
> register callbacks in curl to get your answers and expect them to be
> called on a particular thread, you need to run curl in that thread.
>
> The transfer sizes matter as well. If you have many small transfers,
> multiplexing can really help. For large transfers, the gains are
> less. But then again, if you, have a small request in the "middle"
> it will not need to wait on the large ones to complete.
>
> As you can see, it really depends on your application, how it
> processes the responses, what kind of requests it makes and what
> response times it needs.
Thanks, and yes this is a very interesting point. I only implied it
in my original question, but we are using small HTTP range requests
(CURLOPT_RANGE).
The request size depends, but typically you'd see requests ranging
from the qcow2 cluster size (32K) up to the nbdcopy default request
size (256K). In theory it could be [512b..32M].
The idea behind this is that we can boot a machine off a web server,
downloading only the sectors/clusters as needed (with a lot of caching
going on).
In more advanced cases we're able to inspect a remote disk image,
determine which parts are unused (by using filesystem metadata within
the image), and then download only the parts which contain data, even
modifying the disk image filesystem content in flight, and do all of
this while streaming and making no local copies. It's quite an
interesting research area.
Rich.
Date: Thu, 23 Feb 2023 18:12:17 +0000
On Wed, Feb 22, 2023 at 07:23:42PM +0100, Stefan Eissing via curl-library wrote:
> There is no one size fits all here, as Daniel indicates. If you
> register callbacks in curl to get your answers and expect them to be
> called on a particular thread, you need to run curl in that thread.
>
> The transfer sizes matter as well. If you have many small transfers,
> multiplexing can really help. For large transfers, the gains are
> less. But then again, if you, have a small request in the "middle"
> it will not need to wait on the large ones to complete.
>
> As you can see, it really depends on your application, how it
> processes the responses, what kind of requests it makes and what
> response times it needs.
Thanks, and yes this is a very interesting point. I only implied it
in my original question, but we are using small HTTP range requests
(CURLOPT_RANGE).
The request size depends, but typically you'd see requests ranging
from the qcow2 cluster size (32K) up to the nbdcopy default request
size (256K). In theory it could be [512b..32M].
The idea behind this is that we can boot a machine off a web server,
downloading only the sectors/clusters as needed (with a lot of caching
going on).
In more advanced cases we're able to inspect a remote disk image,
determine which parts are unused (by using filesystem metadata within
the image), and then download only the parts which contain data, even
modifying the disk image filesystem content in flight, and do all of
this while streaming and making no local copies. It's quite an
interesting research area.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2023-02-23