On Mon, 18 Mar 2002 theexperts_at_allprodirect.com wrote:

Ok, my reply comes inlined below.

> What I meant is: I am submitting a search to different sites to obtain
> different results from each and display them on my site. Kind of like a
> meta search engine. I need the remote hosts to race each other to see who
> can be the first one to give my page some content! Not only will the
> content start appearing on my page faster, but oodles of time will be saved
> over the entire process.
>
> Currently, my script must wait for the first site to finish responding
> before moving on to the second, etc. Say, for example, site 1 takes a long
> time to respond. You are staring at a blank screen until it finishes. Then
> you have to wait for site 2, etc. Meanwhile, site 3 might have finished
> much more quickly than either of them. So I would like to submit all the
> requests, let them run concurrenlty on the remote servers, and then display
> the results as they come in.
>
> I know how I would do this without curl: I would send the request to each
> host before looking at any of the responses. Then I would loop through all
> of them looking for responses, reading a chunk of any response, and if the
> end of file for any host is reached I would format and output the results
> for that host. Continuing the loop until all hosts are done.
>
> The question is how to do it with curl. I.e. can I tell curl to submit the
> requests but don't wait for results, then cycle through the hosts to see if
> any of them responded yet, and if so read the result in and process it?
>
> Misc. background: Each site has different requirements, e.g. fieldnames,
> authentication, etc., so the request is formatted as appropriate for each
> site, and does not simply execute the same curl function on a different
> host.
>
> Do you grok my application now?Q ;>)

I understand this perfectly well now. ;-)

The funny thing is, my reply won't be very different! ;-) Ok, skip the
CURLOPT_RANGE talk since that's not what you want.

libcurl supports multiple concurrent transfers in two ways:

1. Run multiple threads, one curl handle and curl_easy_perform() in each
   thread.

2. Use the multi interace, add all transfers to the multi handle and have
   them transfered simultaneously using a single thread.

Approach 1 has been tried numerous times and is very well tried. Approach 2
is much less tested, but a generally nicer way to do this. It is also much
more portable.

The docs for the multi interface is still only present in CVS, due to appear
as web pages any day now...

-- 
    Daniel Stenberg -- curl groks URLs -- http://curl.haxx.se/