I'm new to libcurl and network programming, so I think there are some techniques of which I'm not aware. I did some searching, though, and didn't find anything useful so far, so any advices are much appreciated.
So, I have a program written in C++ with libcurl, that basically crawls the web, downloading the pages. I also have a pretty good internet connection, which easily gives me 4-5 MB/s download speed. However, I can't squeeze any decent network utilization out of my program. Downloading a single page at a time, I rarely see speeds in excsess of 300-500 KB/s. The only idea I have so far is to download multiple pages from different servers simultaneously from multiple threads. I've implemented that, but network utilization graph still doesn't look good: it has rare short spikes up to 1.5-2 MB/s with 0 in between the spikes. I've tried different number of threads, from 5 to 15, but didn't see any visible changes whatsoever.
What I would like to see is network bandwidth utilization of >2 MB/s on average, not only in short spikes. How can I achieve that? I know my connection can handle that, and I don't need for simultaneously downloading pages to be from the same site (server).

Thanks in advance!
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2011-12-17