curl-library
Re: Threaded download slows down after a while
Date: Wed, 3 Sep 2008 09:28:07 +0200 (CEST)
On Tue, 2 Sep 2008, Antonio Weber wrote:
> I've a problem when downloading a lot of similar pages. The attached code 
> runs about 50 times parallel (using pthreads). It should download a html 
> page from which I have to parse descriptions to update a database (about 
> 50000). The download-threads work well for a couple minutes (downloading 
> with full bandwidth usage) but then it slows down and in the end no more 
> traffic can be monitored.
[...]
> So my question, is there an error the way I use libcurl, which causes this 
> behavior?
I wouldn't even know how to do this on purpose.
libcurl has no thread-related code at all internally. Each libcurl transfer is 
completely independent of anything going on in other threads. I don't see how 
libcurl can do anything that slows down other threads, or what other threads 
could do to slow down a particular libcurl thread... :-/
-- / daniel.haxx.seReceived on 2008-09-03