also hey you might want to profile things to see where your bottle necks are
before attempting a solution.

like for instance, if your hard drive is the slow part, and it's 100%
utilized, adding more threads isnt going to make the read head move any
faster.

just being able to see if it's CPU, storage access, or network that is the
bottleneck, and the characteristics of the bottleneck (ie spikes vs
sustained bottleneck) should help you formulate a better solution if you are
really looking to maximize your perf.

On Thu, Aug 11, 2011 at 11:29 AM, Alex Loukissas <alex_at_maginatics.com>wrote:

> On Thu, Aug 11, 2011 at 10:18 AM, Daniel Stenberg <daniel_at_haxx.se> wrote:
> > On Thu, 11 Aug 2011, Alex Loukissas wrote:
> >
> >>> In many situations you won't gain any performance by doing parallell
> >>> uploads to a single server, and you'll get a simpler implementation by
> doing
> >>> serial uploads so I'd recommend that. At least for a first shot.
> >>>
> >>> Just make sure you re-use the same CURL handle _without_ doing
> >>> cleanup/init between each individual transfer.
> >>
> >> Seems like a good plan of attack to use the multi interface as a first
> >> shot.
> >
> > Well, I meant to use the easy interface single-threaded as a first
> attempt,
> > but by all means use the multi interface!
> >
>
> I did try reusing the same handle, which (as expected) had a
> tremendous positive effect in the number of TIME_WAIT sockets and the
> initial tests on performance show some improvement (although not
> drastic). Is there a benefit over what I'm doing now (i.e. looping
> through the URIs and issuing a curl_easy_perform) versus having a
> number of handles in a curl_multi_handle? From what I understand, the
> multi interface is doing the same thing (i.e. serially doing a
> curl_easy_perform for each handle).
>
> >> Another option would be HTTP pipelining but from what I see it's not in
> >> its most mature stage in libcurl. The reason I'm considering pipelining
> is
> >> that the files I'm up/downloading are small size (few to couple hundred
> KB)
> >> and I'm guessing that squeezing many HTTP requests in a single round
> trip
> >> I'll gain some performance. Any thoughts on that?
> >
> > It is not as mature, no, but perhaps more importantly it is only enabled
> for
> > downloading. If you use the multi interface, you can easily just enable
> > pipelining and see how it works/performs.
>
> Great thanks. Reads are more important than writes, so that's OK. I'll
> give it a shot.
>
> >
> > --
> >
> > / daniel.haxx.se
> > -------------------------------------------------------------------
> > List admin: http://cool.haxx.se/list/listinfo/curl-library
> > Etiquette: http://curl.haxx.se/mail/etiquette.html
> >
>
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-library
> Etiquette: http://curl.haxx.se/mail/etiquette.html
>

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2011-08-11