cURL / Mailing Lists / curl-library / Single Mail

curl-library

libcurl 7.9.8: Bottleneck on multhithread HTTP fetches?

From: Andrew Fuller <andrewfuller2k2_at_yahoo.co.uk>
Date: Mon, 13 Oct 2003 17:30:57 +0100 (BST)

Hi,

First of all, thank you for a fab piece of software
which I've been using for a long time without any
problems.

I've been running some loadtesting on my application
and I've narrowed a high-load bottleneck down to curl.
When a large number of threads (>50 for example)
attempt to HTTP GET a document simultaneously, the CPU
on my computer maxes out for a few minutes.

I've turned on CURLOPT_VERBOSE as per the docs, and I
find : Connection #0 seems to be dead. Could somebody
help explain this to me please? My first reaction was
that my server was being overloaded, but I've
eliminated this possibility.

I've created a test application which mimics this
behaviour, please see attached cpp file. It's an
extension of
http://curl.haxx.se/lxr/source/docs/examples/multithread.c
(except it uses Windows threading, sorry!).

Please also find attached results.txt, which shows the
output of the program above with 50 threads, 2 fetches
per thread.

If it's of any use, I ran the program with the same
arguments through Rational Quantify. It identified
the longest path (in terms of time) as
curl_easy_perform()->ws2_32::getsockopt()->ntdll::NtRemoveIoCompletion()

Of course, I understand that there'll be an inherent
delay when 50 curl instances attempt to grab a
connection simultaneously! However, I'm concerned
about the high CPU usage and the delays associated
with it. It seems to me from curl's verbose output
(see results.txt) that it only ever uses one
connection (#0) - is this correct? Is the approach in
my code incorrect?

Here's my setup information :
libcurl 7.9.8
windows 2000 professional
MSVC++ 6.0
{server : Apache 2.0.47 (win32) }

Thanks for reading this far :) Any suggestions would
be appreciated.

regards
Andrew Fuller

________________________________________________________________________
Want to chat instantly with your online friends? Get the FREE Yahoo!
Messenger http://mail.messenger.yahoo.co.uk

// CurlTest.cpp : Defines the entry point for the console application.
// Requires _MT to be defined to compile

#include <curl/curl.h>
#include <process.h>
#include <stdlib.h>

// wait for many threads to finish
bool WaitForThreadsToFinish(const HANDLE* pHandles, const int nNumHandles);

// executes in own thread : does a curl fetch
unsigned long __stdcall pull_one_url(void* pArg)
{
        CURL* pHandle = (CURL*)pArg;
    if(pHandle)
        {
                curl_easy_perform(pHandle);
        }
        return 1;
}

// main
int main(int argc, char* argv[])
{
        // check args
        if(argc < 4)
        {
                printf("\nUsage : %s num_threads fetches_per_thread uri_to_fetch\n\n", argv[0]);
                return 1;
        }
        int nNumThreads = atoi(argv[1]);
        int nNumFetchesPerThread = atoi(argv[2]);

// create handles
CURL** pHandles = new CURL*[nNumThreads];

        // init
        for(int i = 0; i<nNumThreads; i++)
        {
                pHandles[i] = curl_easy_init();
                curl_easy_setopt(pHandles[i], CURLOPT_VERBOSE, 1);
                curl_easy_setopt(pHandles[i], CURLOPT_URL, argv[3]);
        }

        // want each thread to fetch nNumFetchesPerThread times
        for(int j = 0; j < nNumFetchesPerThread; j++)
        {
                printf("\n\n***\nStarting fetch #%d/%d on %d threads\n\n", j, nNumFetchesPerThread, nNumThreads);

// create holder for thread handles
HANDLE* pThreadHandles = new HANDLE[nNumThreads];

                // send fetches
                for(i = 0; i<nNumThreads; i++)
                {
                        pThreadHandles[i] = CreateThread(0, 0, pull_one_url, pHandles[i], 0, 0);
                }

                // wait for completion
                if(!WaitForThreadsToFinish(pThreadHandles, nNumThreads))
                {
                        printf("Wait #%d failed (%d)!!\n", j, GetLastError());
                        break;
                }

                // clean thread handles
                delete[] pThreadHandles;
        }

        // cleanup
        for(i = 0; i<nNumThreads; i++)
        {
                curl_easy_cleanup(pHandles[i]);
        }

// clean curl handles
delete[] pHandles;

return 0;
}

// wait for many threads to finish
bool WaitForThreadsToFinish(const HANDLE* pHandles, const int nNumHandles)
{
int i = nNumHandles / MAXIMUM_WAIT_OBJECTS;
int j = nNumHandles % MAXIMUM_WAIT_OBJECTS;

        for(int k = 0; k < i; k++)
        {
                if(WAIT_FAILED == WaitForMultipleObjects(MAXIMUM_WAIT_OBJECTS, pHandles + (k * MAXIMUM_WAIT_OBJECTS), TRUE, INFINITE))
                {
                        return false;
                }
        }

        if(j>0)
        {
                if(WAIT_FAILED == WaitForMultipleObjects(j, pHandles + (i * MAXIMUM_WAIT_OBJECTS), TRUE, INFINITE))
                {
                        return false;
                }
        }

// success
return true;
}

{arguments : 50 2 http://10.0.0.180/test.txt}

* About to connect() to 10.0.0.180:80
* About to connect() to 10.0.0.180:80
* Connected to 10.0.0.180 (10.0.0.180) port 80
> GET /test.txt HTTP/1.1

Host: 10.0.0.180

Pragma: no-cache