On Tue, 25 Mar 2008, Ramprakash Jelari thinakaran wrote:

> Given a 10 GB data, what could be the performance difference or any other
> difference between the following two approaches?
>
> 1. Give the 10 GB data to libCurl and do a single post, and libCurl will
> take care of chunking the data and send to server.
> 2. Split 10GB into 1KB of chunk and do post for each 1KB data.
> [ I hope doing multiple post still continue with the same connection,
> as well as the only over head i see for the '2'nd step
> is adding header information each time the chunk is passed ]
>
> I have a background architecture which can support the step(2). But what i
> want to know is what could the noticable defect in it over step (1).?

The answer would rather depend on a lot of factors you left out in this
description.

How likely is the connection to disconnect before the 10GB is done? How much
is the penalty for retrying either scenario after a connection break?

In either case libcurl won't even have to "chunk" the data, if you know the
full size before the transfer starts.

The case #2 of course have a (much) larger overhead since it'll add the HTTP
request-headers and get HTTP responses for every 1KB data sent instead of just
sending and getting them once for 10GB. Of course, with a decent HTTP 1.1
server all posts in #2 will re-use the same connection. But note that unless
you do pipelining you also limit the throuput quite dramatically since you can
only send 1KB before having to wait for the response.

If you would do #2, I would suggest a much larger chunk size than a mere
1KB... Of course that also depends on your network bandwidth etc.

-- 
  Commercial curl and libcurl Technical Support: http://haxx.se/curl.html