On Mon, 15 Jan 2007 12:08:37 -0600, Daniel Stenberg <daniel_at_haxx.se> wrote:

> On Sun, 14 Jan 2007, Justin Graves wrote:
>
>> In Windows, the app flies. It runs very fast and very solid. In
>> Linux, however, curl_easy_perform seems to take far longer, and the
>> more threads I make it seems to get even worse. If I use roughly 15 or
>> more concurrent threads, curl starts failing transfers a lot for no
>> apparant reason.
>
> I've never heard of that before. "no apparant reason" you say, so what
> exactly does happen?

Well, I set up the curl handle as mentioned, then I call curl_easy_perform
and it sits in that function, mostly idle CPU-wise, and just takes a while
to return.

> And in general it sounds like an over-use of threads that in many ways
> will prevent libcurl from performing at maximum speed, but then of
> course you do it the same way on both operating systems.

I'm not sure why the threads would be over-used? I have a whole lot of
URLs I need to pull, so I use multiple threads to do it in less time.

>> This is identical to the operation in Windows, but for some reason it
>> runs much quicker there.
>
> The effective code in libcurl differ very little between Windows and
> Linux so I would guess that there are external explanations to the
> differences, i.e your network, kernel, machine conditions or similar.

This is possible, I suppose. I have another Linux machine I may be able
to test on that is on a T3 connection. I'll test it on that perhaps
tomorrow. Locally, though, they are on an identical network, just
connected to two different ports on the same router. I suppose there
could be some configuration issue on the Linux machine, but its really
mostly just a base Ubuntu install.

>> The delay seems to be in curl_easy_perform. It doesn't use CPU (the app
>> will sit at 0-3% cpu while running).
>
> So what happens on the network? It sounds like libcurl is waiting for
> action...

This is strange, I just monitored it for a bit. It looks like it sends
out all the requests then just idles with no incoming traffic for a few
seconds, then it starts coming in. Same sites on Windows have far less
delay. I suppose this must be a machine configuration issue.

Thank you much for your input.

- Justin
Received on 2007-01-15