On Sun, 18 Dec 2011, Alex wrote:

> Thanks for your suggestions. I don't think there is any problem, just an
> inefficiency. I mean, every single page's speed might be limited by the
> server. Besides, pages are small (160 KB on average, based on 70 000 pages
> crawled so far), so overhead of resolving DNS names, creating thread,
> opening socket and so on is significant.

Yes, you're right. If you want to optimize for some kind of transfer
performance you should probably consider what to do to minimize the impact of
all those details. Such as perhaps not dynamically start new threads or even
use less threads.

> Does curl tool implement any kind of parallel download?

Nope. It may one day since libcurl provides an API for it, but it's not being
worked on.

> I currently use "easy" interface, because it really is very easy to use. I
> took a look at "multi" documentation, but didn't understand how to use it.
> Besides, I really like to manage threads myself, so that I understand what's
> going on inside my program. Will multi interface be more efficient than my
> naive approach?

If you're aiming at a very large amount of parallel transfers, if you rather
want an non-blocking API etc then the multi interface is your friend.

We basically provide a bunch of different ways to do transfers, but you get to
pick which to use yourself!

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html