cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: Threaded download slows down after a while

From: Daniel Stenberg <daniel_at_haxx.se>
Date: Wed, 3 Sep 2008 09:28:07 +0200 (CEST)

On Tue, 2 Sep 2008, Antonio Weber wrote:

> I've a problem when downloading a lot of similar pages. The attached code
> runs about 50 times parallel (using pthreads). It should download a html
> page from which I have to parse descriptions to update a database (about
> 50000). The download-threads work well for a couple minutes (downloading
> with full bandwidth usage) but then it slows down and in the end no more
> traffic can be monitored.

[...]

> So my question, is there an error the way I use libcurl, which causes this
> behavior?

I wouldn't even know how to do this on purpose.

libcurl has no thread-related code at all internally. Each libcurl transfer is
completely independent of anything going on in other threads. I don't see how
libcurl can do anything that slows down other threads, or what other threads
could do to slow down a particular libcurl thread... :-/

-- 
  / daniel.haxx.se
Received on 2008-09-03