cURL / Mailing Lists / curl-library / Single Mail

curl-library

RE: Curl downloading timeouts/speed issues with many concurrent downloads?

From: Michael Mastroianni <MMastroianni_at_choicestream.com>
Date: Tue, 19 Jul 2005 14:24:36 -0400

You're right, I meant 7.12.3-1.

It seems to only happen if I have at least 10 threads, and it doesn't
always happen, even then, except on one particular machine (I normally
run with hundreds, which works on most of our machines just fine).

Our normal production timeouts are 4 seconds, but I get these results on
this machine even with 20 second timeouts.

Cat'ing /etc/issue gives me this on the machine in question:
Red Hat Linux release 9 (Shrike)

-----Original Message-----
From: curl-library-bounces_at_cool.haxx.se
[mailto:curl-library-bounces_at_cool.haxx.se] On Behalf Of Daniel Stenberg
Sent: Tuesday, July 05, 2005 6:22 PM
To: libcurl development
Subject: Re: Curl downloading timeouts/speed issues with many concurrent
downloads?

On Tue, 5 Jul 2005, Michael Mastroianni wrote:

> I'm using 12.3.1

Wow, you must be living in a distant future! 7.14.0 is the most recent
released version...

I figure you mean 7.12.3-1 or something?

> I have an application which concurrently downloads many urls using the
easy
> interface on lots of threads (for various reasons, the multi interface
isn't
> something I can easily accommodate.). Anyway, on one of our linux
boxes, my
> application times out 50% of the time, and gets DNS failures 21% of
the time
> (this is pretty rare on our windows boxes). But if I wget these urls
on the
> command line on the machine, they download just fine.

You have any timeouts set in libcurl? If so, for how long?

What if you strace or similar (possibly even ethereal), would does that
show
that libcurl does or doesn't do in the failing cases?

Are the failures evenly spread out over time? Do you get the problems
even if
only one thread is running? If not, how many threads are required?

If you must use many threads to see the problem, do the problems still
occur
if you slow down the rate of their operations?

> I'm trying to hunt down what flavor of Red Hat the thing is running.

I don't think that's very important. (but cat /etc/issue usually tells)

> Does this ring a bell with anyone? Has anyone else reported such
behavior?

Not really and no. :-(

-- 
  Commercial curl and libcurl Technical Support:
http://haxx.se/curl.html
Received on 2005-07-19