Hi Daniel,

Thanks for the reply. Couple comments inline.

On Thu, Aug 11, 2011 at 3:01 AM, Daniel Stenberg <daniel_at_haxx.se> wrote:
> On Wed, 10 Aug 2011, Alex Loukissas wrote:
>
>> I am looking to refactor this code now, so that I get a speedup by
>> parallelism (the order that these requests are performed doesn't need to be
>> FIFO or anything, as long as they all succeed), and connection reuse (i.e.
>> filling up the pipe).
>
> In many situations you won't gain any performance by doing parallell uploads
> to a single server, and you'll get a simpler implementation by doing serial
> uploads so I'd recommend that. At least for a first shot.
>
> Just make sure you re-use the same CURL handle _without_ doing cleanup/init
> between each individual transfer.

Seems like a good plan of attack to use the multi interface as a first
shot. Another option would be HTTP pipelining but from what I see it's
not in its most mature stage in libcurl. The reason I'm considering
pipelining is that the files I'm up/downloading are small size (few to
couple hundred KB) and I'm guessing that squeezing many HTTP requests
in a single round trip I'll gain some performance. Any thoughts on
that?

>
>> One of the things I've looked at is going into the curl_multi_* interface,
>> have a set number of handles and reuse them until all requests have been
>> served. My question is the following: is there a benefit of doing this
>> (which from what I understand is single-threaded and serialized --see:
>> http://curl.haxx.se/mail/lib-2011-06/0065.html) over having a thread pool of
>> curl_easy_handles that each has a connection to the server?
>
> For a low degree of parallellism, you probably will not see any speed
> differences between the multi interface and doing threaded easy interface
> transfers. For a large number of simultaneous transfers (where large is at
> least above 100), you should see better performance by going with the
> multi_socket API. Mostly it is a matter of how you want to design your
> software.
>
> --
>
>  / daniel.haxx.se
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-library
> Etiquette:  http://curl.haxx.se/mail/etiquette.html
>

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2011-08-11