Hi libcurl devs,

I'm writing an application that uses libcurl, and I have no prior
expertise with HTTP, so I'd like to make sure I got things right.

I'm working on the internal http client of casync [1]. This client is
simple, it basically has a list of files to download, and its job is to
download it efficiently. We're talking about small chunks of data
(around 64KB), but the list is possibly huge (60,000 chunks is very
possible). And we talk to only ONE server.

Since we live in a modern world, I explicitly enable
`CURL_HTTP_VERSION_2_0` and `CURLPIPE_MULTIPLEX`, and I assume that the
server supports it.

In a **first implementation**, I just create a curl easy handle for each
chunk I need to download (so, possibly 60k easy handles), add it to the
curl multi, and then I let curl deal with it. I also make sure to set
`CURLMOPT_MAX_TOTAL_CONNECTIONS` to ensure that the whole thing doesn't
go crazy (I used 64 at first, but after more reading I wonder if I
should lower that to 8).

It works good this way. Even too good. My issue then, during local tests
(with both client and server on my machine), is that the client isn't
fast enough to handle all the incoming chunks. Indeed, the client needs
to give the chunks to another co-process, through a custom IPC, and this
proved to be the bottleneck. So what happened is that all my chunks were
downloaded very quickly, and then sat in RAM until the client had time
to forward it to its co-process. Even though it works, it possibly uses
a lot of RAM and it's not nice.

Of course, this doesn't happen in "real-life", when the server is away
and the latency is higher. Then the client has time to handle the
chunks, and everything works beautifully.

I didn't find a way to to tell libcurl to pause or slow down in case
things go too fast, so I went for a **second implementation**, slightly
different. I decided that instead of creating one easy handle per chunk
request and feed it all to the curl multi handle, I would only create a
small number of easy handles (let's say 8) and give it to curl multi.
Only when a chunk is downloaded and handled by the client, then I re-use
the easy handle (ie. remove it from the multi handle, set a new URL, and
give it back to the curl multi for processing).

This implementation works good as well.

Now, I take a bit of time to think, and I wonder if this second
implementation is really the smart thing to do. More precisely: by
feeding handles one by one (even though we might have 8 active handles
in curl multi at the same time), do I prevent internal optimization
within libcurl? How can libcurl multiplex efficiently if I don't tell it
in advance the list of chunks I want to download?

So basically, I think that my first implementation was better than the
second one, can you agree or disagree, based on your knowledge of
libcurl internals?

I also take this chance to ask a second question, out of curiosity: with
HTTP/2 multiplex enabled, will libcurl also attempt to open concurrent
connections, and do multiplex on all these connections? Or does it stick
to one connection?

Thanks!

  Arnaud

----
[1]:
http://0pointer.net/blog/casync-a-tool-for-distributing-file-system-images.html
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html