On Mon, 13 Oct 2003, Andrew Fuller wrote:

> I've been running some loadtesting on my application and I've narrowed a
> high-load bottleneck down to curl. When a large number of threads (>50 for
> example) attempt to HTTP GET a document simultaneously, the CPU on my
> computer maxes out for a few minutes.

I don't understand how curl would behave differently depending on the amount
of threads. If you have 50, 200 or 10, curl will behave the same.

You're also using a version that is many releases old that we have fixed
numerous bugs in since its release. Have you verified if a later version makes
any difference?

> I've turned on CURLOPT_VERBOSE as per the docs, and I find : Connection #0
> seems to be dead. Could somebody help explain this to me please? My first
> reaction was that my server was being overloaded, but I've eliminated this
> possibility.

If you get a more recent version and you get the same output, then I would be
interested in digging into this. curl 7.9.8 is history to me, and I don't care
very much about possible bugs in it. Sorry, but my time is too limited for
that.

> If it's of any use, I ran the program with the same arguments through
> Rational Quantify. It identified the longest path (in terms of time) as
> curl_easy_perform()->ws2_32::getsockopt()->ntdll::NtRemoveIoCompletion()

I find that very strange.

> Of course, I understand that there'll be an inherent delay when 50 curl
> instances attempt to grab a connection simultaneously! However, I'm
> concerned about the high CPU usage and the delays associated with it. It
> seems to me from curl's verbose output (see results.txt) that it only ever
> uses one connection (#0) - is this correct? Is the approach in my code
> incorrect?

If all threads do two curl_easy_perform() to the same host, then both calls
will most likely use the same connection, so yes, then the second one would
re-use connection #0.

-- 
 Daniel Stenberg -- curl: been grokking URLs since 1998
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php