1. Dan, thanks for pointing out the global and cleanup missing -- I've
added those - but it didn't improve things.
2. I don't believe the latency is expected to be increasing with the
number of threads unless that number threatens to hog one/ or more
resources( but in my case the process takes 0.5% of the cpu, and the
data is ~15k per call, so 100*15 = 1.5Mb -- my connection is 10M , so
I doubt it is blocking on that ). However I am not sure if the threads
would be scheduled like ordinary processes, or they would be scheduled
to run within the time slice of the parent process . If the latter is
true, then it would sort of help explain what I am seeing.

Radu

On Sat, Apr 30, 2011 at 3:00 PM, Ben Noordhuis <info_at_bnoordhuis.nl> wrote:
> On Sat, Apr 30, 2011 at 22:49, Daniel Stenberg <daniel_at_haxx.se> wrote:
>> It would make sense if the 10 threads used 100% CPU, but usually getting 10
>> simultaneous transfers don't consume all CPU so adding more threads should
>> not make them run (much) longer.
>
> The more threads you have, the more time they'll spent in a blocked
> 'I/O pending' state. And a blocked thread won't immediately wake up
> when new data arrives.
>
> Back-of-the-envelope example: assume 100 threads with a 10 ms slice
> each, single-core machine.
>
> When thread 0 attempts a read() that blocks, threads 1-99 run. Let's
> say the expected data arrives 50 ms after thread 0 blocked - thread 0
> still has to wait another 950 ms before it runs again. Repeat this a
> couple of times and you'll see why latency grows relative to the
> number of threads.
>
> Reality is of course not so clear-cut - no modern operating system
> does strict round-robin scheduling - but that doesn't take away from
> the general validity of my argument.
>
> Tanenbaum's "Operating Systems: Design And Implementation" has a
> thorough and well-written chapter on schedulers, if you're interested.
> It's a great book all around actually, I can really recommend it.
>
> Anyway, this is not a bug in curl - it's how the scheduler works.
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-library
> Etiquette:  http://curl.haxx.se/mail/etiquette.html
>

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2011-05-01