curl-library
Re: Threaded download slows down after a while
Date: Wed, 3 Sep 2008 09:28:07 +0200 (CEST)
On Tue, 2 Sep 2008, Antonio Weber wrote:
> I've a problem when downloading a lot of similar pages. The attached code
> runs about 50 times parallel (using pthreads). It should download a html
> page from which I have to parse descriptions to update a database (about
> 50000). The download-threads work well for a couple minutes (downloading
> with full bandwidth usage) but then it slows down and in the end no more
> traffic can be monitored.
[...]
> So my question, is there an error the way I use libcurl, which causes this
> behavior?
I wouldn't even know how to do this on purpose.
libcurl has no thread-related code at all internally. Each libcurl transfer is
completely independent of anything going on in other threads. I don't see how
libcurl can do anything that slows down other threads, or what other threads
could do to slow down a particular libcurl thread... :-/
-- / daniel.haxx.seReceived on 2008-09-03