I have a program which fetches URLs from a MySQL table and spawns threads
to retrieve those URLs using libCurl. The app is cross-platform in
Windows and Linux. The only major difference is, of course, Windows using
Win32 threads and Linux using pthreads.

In Windows, the app flies. It runs very fast and very solid. In Linux,
however, curl_easy_perform seems to take far longer, and the more threads
I make it seems to get even worse. If I use roughly 15 or more concurrent
threads, curl starts failing transfers a lot for no apparant reason.

The app works as such:
- curl_global_init called at application start
- Retrieve 100 URLs from MySQL
- For each URL while a thread is available, spawn thread to fetch URL (if
no threads are available, main thread sleeps for 250ms then checks again)
- Repeat

In each thread, a curl handle is made with curl_easy_init. It is set up
as such:
curl_easy_setopt(pHandle, CURLOPT_WRITEFUNCTION, CURLWrapWriteData);
curl_easy_setopt(pHandle, CURLOPT_WRITEDATA, this);
curl_easy_setopt(pHandle, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(pHandle, CURLOPT_MAXREDIRS, 5);
curl_easy_setopt(pHandle, CURLOPT_NOSIGNAL, 1);
curl_easy_setopt(pHandle, CURLOPT_NOPROGRESS, 1);
curl_easy_setopt(pHandle, CURLOPT_DNS_CACHE_TIMEOUT, 0);
curl_easy_setopt(pHandle, CURLOPT_DNS_USE_GLOBAL_CACHE, 0);

Then:
if(curl_easy_setopt(pHandle, CURLOPT_URL, szURL))
        return false;

if(curl_easy_perform(pHandle))
        return false;

After that, some string work is done on the retrieved data and
curl_easy_cleanup is called on the handle, then that thread dies.

This is identical to the operation in Windows, but for some reason it runs
much quicker there. I will note:
The delay seems to be in curl_easy_perform. It doesn't use CPU (the app
will sit at 0-3% cpu while running). I have tested this using the same
URL batch on both platforms. Both machines are about the same speed and
on the same network hooked up to the same switch/router.

Windows is using the downloaded MSVC binaries, version 7.16.0. Linux was
using 7.15.4, but I just built c-ares and built libCurl from source, so it
is now 7.16.0 as well, but the problem persists. Linux OS is Ubuntu 6.10
(Edgy Eft; kernel 2.6.17, gcc/g++ version 4.1.2).

Any suggestions on this?

Thanks for any help.

- Justin
Received on 2007-01-15