>Hello, lemme share my experiences with libcurl as I've also needed
maximum performance while downloading multiple urls a while ago.
Thank you for sharing your experiences.

>First, regarding your question, as Daniel said, if you can't build curl
to use asynchronous resolver, running a threaded application will of
course result in better performance as it can run same amount of
resolves in parallel as your thread count.
I did not aware that name resolver would make difference. But now, after
this email, I built libCURL to make use of c-ares library for name
resolution. Another thing I would like to brought up here is, I use IP
address in my HTTP request (for e.g.,
http://190.190.190.190:85/get_me_this_file.jpeg). Is there a way to
optimize here like bypassing name resolution!!! I am not sure whether
this is the right question to ask or I do not know if this is even
possible?

>However, if you're able to run with asynchronous resolver like c-ares,
please note that while c-ares does perform async resolve which should
greatly improve speed when using single thread and multi inferface, it
is (Daniel isn't gonna like this comment :)) a fairly slow library. In
fact it's so slow that it'd most likely take longer than 1.75 secs to
download 1500 urls if you create an easy handle for each url and add it
to multi interface.

You are right. I am creating one easy handle for url and adding to multi
interface. And each thread will have its own instance of multi handle
meaning, multi handle is not shared across threads. How are you handling
easy handles? I mean, new_url->easy_handle or how?

>For my application (whose requirements are to do about 400 http reqs
per sec 24/7) I'm using 8 threads (got 8 cpu cores), each one has its
own multi interface and I'm creating and adding easy handles to multi
interface on round robin basis becase when using libcurl/c-ares combo
with multi interface in only one thread app wasn't able to keep up.

My experiences are based on libcurl/7.21.4 and c-ares/1.7.4 and I
haven't been really following release notes of both libraries as current
setup satisfied my needs (don't try to fix things that work :)) so I'm
sorry for bad mouthing the libs if these things were fixed, but unless
libcurl went under considerable changes of how it uses c-ares (based on
5 second look of Curl_open() it appears it didn't) there shouldn't be
any performance differences between 7.21.4 and 7.23.1).

I am using curl 7.21.7 and c-ares 1.7.5.

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2011-11-22