cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: Simultaneous requests in multi-threaded application

From: Daniel Stenberg <daniel_at_haxx.se>
Date: Tue, 1 Apr 2008 09:57:27 +0200 (CEST)

On Mon, 31 Mar 2008, Athari wrote:

> I am developing a crawler which will download files simultaneously, and
> users need to be able to control each thread, i.e. abort, pause (?), see
> progress. DNS cache and cookies need to be shared between the handles, and
> cookies need to be preserved between program runs. What is the most adequate
> (read: easy) way to do this?

This can be done using the easy interface and multiple threads, or using the
multi interface in a single thread (with some caveats).

> At first I thought to create necessary number of easy cURL handles sharing
> data using share interface, but this way I will not be able to control
> downloads.

Cannot control them in what way?

> Plus, I did not understand when CURLSHOPT_LOCKFUNC is needed - only when I
> use a share handle from multiple threads, or when multiple easy handles are
> used simultaneously?

If you're sharing a share "object" from multiple simultaneous threads you need
it. If you're carefully only using one at a time you won't need it.

-- 
  Commercial curl and libcurl Technical Support: http://haxx.se/curl.html
Received on 2008-04-01