Hi Justin,

On 1/15/07, Justin Graves <justin_at_inventedexistence.com> wrote:

> The app works as such:
> - curl_global_init called at application start
> - Retrieve 100 URLs from MySQL
> - For each URL while a thread is available, spawn thread to fetch URL (if
> no threads are available, main thread sleeps for 250ms then checks again)
> - Repeat
>
> In each thread, a curl handle is made with curl_easy_init. It is set up
> as such:
> curl_easy_setopt(pHandle, CURLOPT_WRITEFUNCTION, CURLWrapWriteData);
> curl_easy_setopt(pHandle, CURLOPT_WRITEDATA, this);
> curl_easy_setopt(pHandle, CURLOPT_FOLLOWLOCATION, 1);
> curl_easy_setopt(pHandle, CURLOPT_MAXREDIRS, 5);
> curl_easy_setopt(pHandle, CURLOPT_NOSIGNAL, 1);
> curl_easy_setopt(pHandle, CURLOPT_NOPROGRESS, 1);
> curl_easy_setopt(pHandle, CURLOPT_DNS_CACHE_TIMEOUT, 0);
> curl_easy_setopt(pHandle, CURLOPT_DNS_USE_GLOBAL_CACHE, 0);
>
> Then:
> if(curl_easy_setopt(pHandle, CURLOPT_URL, szURL))
> return false;
>
> if(curl_easy_perform(pHandle))
> return false;
>
> After that, some string work is done on the retrieved data and
> curl_easy_cleanup is called on the handle, then that thread dies.
>
>
> This is identical to the operation in Windows, but for some reason it runs
> much quicker there. I will note:
> The delay seems to be in curl_easy_perform. It doesn't use CPU (the app
> will sit at 0-3% cpu while running). I have tested this using the same
> URL batch on both platforms. Both machines are about the same speed and
> on the same network hooked up to the same switch/router.
>
> Windows is using the downloaded MSVC binaries, version 7.16.0. Linux was
> using 7.15.4, but I just built c-ares and built libCurl from source, so it
> is now 7.16.0 as well, but the problem persists. Linux OS is Ubuntu 6.10
> (Edgy Eft; kernel 2.6.17, gcc/g++ version 4.1.2).
>
> Any suggestions on this?

Thread per-request was a common design in 80-es of the last century.
Y can do all in one thread, using e.g. curl multi interface.

If you are still need threads, use a pool of threads prepared in advance, so
that their creation will not be an issue.

Another issue may be the DNS lookup time. Look in your sniffer, if this
lookup is slower for your linux. If yes, please look into various libcurl
DNS options, but take care about the thread-safety.

-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...................................................................
Navigare necesse est, vivere non est necesse
...................................................................
http://sourceforge.net/projects/curl-loader
A powerful open-source HTTP/S, FTP/S traffic
generating, loading and testing tool.