curl-library
Re: Please advise on how to improve libcurl HTTP GET performance
From: Alex <Alexxxx89_at_ya.ru>
Date: Tue, 11 Oct 2011 00:23:24 +0400
Date: Tue, 11 Oct 2011 00:23:24 +0400
1) No way, I'm downloading to a memory buffer.
2) I didn't try telnet, but both Opera and console CURL tool download the very same page at ~300 KB/s, which is much closer to my actual net speed. Just a reminder: libcurl gives me 25-40 KB/s.
Here's the code, you can see what page I'm testing on.CByteArray is just a wrapper for void* buffer, it's append uses realloc and memcpy.
size_t onDownloadCallback( char* ptr, size_t size, size_t nmemb, void* receiver )
{
size_t actualSize = size * nmemb;
CByteArray* dest = (CByteArray*) receiver;
if (!dest)
return 0;
dest->append(ptr, actualSize);
return actualSize;
}
void main ()
{
CURL *curl; CURLcode res;
CHtmlPage page (CUrl(""));
curl = curl_easy_init();
time_t start,stop;
if(curl) {
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &page.byteArray());
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, onDownloadCallback);
curl_easy_setopt(curl, CURLOPT_URL, "http://www.math.utah.edu/~pa/math/largeprime.html");
start = clock();
res = curl_easy_perform(curl);
stop = clock();
{
size_t actualSize = size * nmemb;
CByteArray* dest = (CByteArray*) receiver;
if (!dest)
return 0;
dest->append(ptr, actualSize);
return actualSize;
}
void main ()
{
CURL *curl; CURLcode res;
CHtmlPage page (CUrl(""));
curl = curl_easy_init();
time_t start,stop;
if(curl) {
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &page.byteArray());
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, onDownloadCallback);
curl_easy_setopt(curl, CURLOPT_URL, "http://www.math.utah.edu/~pa/math/largeprime.html");
start = clock();
res = curl_easy_perform(curl);
stop = clock();
curl_easy_cleanup(curl); }
cout << "Download speed: " << page.byteArray().getSize() * 1000 / ((stop - start) * 1024) << " KB/s\n";
}
cout << "Download speed: " << page.byteArray().getSize() * 1000 / ((stop - start) * 1024) << " KB/s\n";
}
The code still looks perfectly OK to me, don't understand what's the bottleneck.
Regards,
Alex
10.10.2011, 20:51, "Alan Wolfe" <alan.wolfe@gmail.com>:
#1 - Does your program block at all on disk i/o (are you reading or writing to the disk?) if so that could be bottlenecking your D/L.
#2 - Are you sure the problem is on your end, not the server side?
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2011-10-10