On Tue, 3 May 2016, Alexandre BOUIN wrote:

> For our project, I would like to split a download through a single
> connexion, using Pipelining. For the moment, I succeed in making a loop that
> send x range requests, and send next range request as soon a previous one
> has finished. x is the number of the pipeline length.

My main question is: why? The only reason to do pipelining is to overcome
latency when using HTTP/1.1 - the time from the end of the previous transfer
until the next one starts. You're downloading a single resource, so do
pipelining for a single file just adds a lot of complexity with no added
benefits.

Then, if you're trying pipelining over the Internet against servers you don't
yourself control or limit your operations against a tested set, you're up for
a world of fun when middle boxes and servers can disagree on how pipelining is
supposed to work... If at all possible, consider going HTTP/2 instead.

Or is there a chance that you're actually talking about doing N transfers in
parallel using N connections?

> My concern is that I'm not able for the moment to reach full throttle, since
> throughput is limited to let say 33% of the real bandwidth, with 4
> pipelines.

"4 pipelines" means that one transfer is going and 3 are waiting to get
served. There is only one tranfer going on at the time with pipelining.

> If I increase pipeline length (up to 50) or I increase max host connexion
> (up to 3-4), I'm able to reach a better throughput, but my feeling is that
> too many requests are used for the benefits.

Increasing pipeline length simply cannot increase throughput. If that still
happens when you increase the length then something really magic and evil is
interfering and affecting your traffic that goes beyond what the client
controls.

Increasing the number of connections can increase throughput is several ways:
primarily it gives you more early bandwidth during the TCP slow start period
as when the connection(s) go full speed there should be little to no
difference between N connections and 1, but there might still be a difference
thanks to traffic shaping and more on the route. Packet loss situations may
however impact multi connection situation worse than single connection
transfers. It will of course depend on the specific situations.

> Could you please help me understanding the reasons why I cannot reach the
> reference throughput using a reasonable pipeline length ?

Can you instead explain to us why it pipeline length would change the speed at
all in the first place?

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:  https://curl.haxx.se/mail/etiquette.html