Gary,

Thanks for your suggestions. I don't think there is any problem, just an inefficiency. I mean, every single page's speed might be limited by the server. Besides, pages are small (160 KB on average, based on 70 000 pages crawled so far), so overhead of resolving DNS names, creating thread, opening socket and so on is significant.
Does curl tool implement any kind of parallel download?

P. S. I currently use "easy" interface, because it really is very easy to use. I took a look at "multi" documentation, but didn't understand how to use it. Besides, I really like to manage threads myself, so that I understand what's going on inside my program. Will multi interface be more efficient than my naive approach?
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2011-12-17