cURL / Mailing Lists / curl-library / Single Mail

curl-library

Pipelined download in my lib stopped working probably since libcurl 7.30

From: Xavi Artigas <xartigas_at_fluendo.com>
Date: Mon, 31 Mar 2014 13:16:51 +0200

Hello!

First, some context:
In my company we use a downloading library, which is just a thin wrapper
around libcurl.
We are interested in pipelining for maximum throughput, but we also need to
abort all pending requests (except the one currently being processed) and
quickly start processing a new set of requests.
Now, if my library receives a list of 10 urls and I pass them all to
libCurl, they will all be immediately requested, and the server will start
answering them in order, giving me full-throughput pipelining. Perfect!
Except that I have no way of cancelling these requests without closing the
TCP connection (which is slow).
What our library does is enqueueing internally the list of requests, and
passes them to libCurl one by one, only when the previous transfer is about
to finish (at 75%, actually). In this way, we still get full pipelining,
and we can easily cancel petitions most of the time, because they have not
been passed to libCurl.

And now the problem:
This setup has been working for a while, and a client or ours has been
using it without problems, using libCurl 7.22 and 7.27, both on Ubuntu and
Windows 7.
Now I have a report from our client, saying that sometimes it receives
corrupted data. He recently updated to libCurl 7.32, and I can reproduce
the issue, using libCurl 7.36.

I found out that a major revamp of the pipelining code in libCurl was made
in 7.30, so the numbers seem to fit.
But given the effort that libCurl puts in maintaining a stable API and ABI,
it seems strange.

So, anyone has any idea I can try?
I am asking here because I would like to avoid having to use WireShark to
track down packet reordering issues...

Thanks a lot!
Xavi

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2014-03-31