cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: maximizing performance when issuing a large number of GET/PUT requests to a single server

From: Daniel Stenberg <daniel_at_haxx.se>
Date: Thu, 11 Aug 2011 19:18:45 +0200 (CEST)

On Thu, 11 Aug 2011, Alex Loukissas wrote:

>> In many situations you won't gain any performance by doing parallell
>> uploads to a single server, and you'll get a simpler implementation by
>> doing serial uploads so I'd recommend that. At least for a first shot.
>>
>> Just make sure you re-use the same CURL handle _without_ doing cleanup/init
>> between each individual transfer.
>
> Seems like a good plan of attack to use the multi interface as a first
> shot.

Well, I meant to use the easy interface single-threaded as a first attempt,
but by all means use the multi interface!

> Another option would be HTTP pipelining but from what I see it's not in its
> most mature stage in libcurl. The reason I'm considering pipelining is that
> the files I'm up/downloading are small size (few to couple hundred KB) and
> I'm guessing that squeezing many HTTP requests in a single round trip I'll
> gain some performance. Any thoughts on that?

It is not as mature, no, but perhaps more importantly it is only enabled for
downloading. If you use the multi interface, you can easily just enable
pipelining and see how it works/performs.

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Received on 2011-08-11