cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: Download performance

From: Jacky Lam <sylam_at_emsoftltd.com>
Date: Wed, 27 Mar 2002 16:59:00 +0800

> > Can I download more than 1 object with the same handle in one thread?
> > (someone is working on that now?)
>
> Yes, you *should* re-use the handle within the same thread as much as
> possible, as that'll increase performance, due to persistant connections
> and handle-oriented caches.
        
>
> > Also, I find that each thread will try to resolve the same host name.
> > Is there any DNS caching in libcurl?
>
> Yes, but it is per handle only, as there's no support for mutexes in
> libcurl. You can force the DNS cache to be used globally, but that will
> only work if you fire off once libcurl thread at a time. Again, since there
> are no mutex support...
>

        I see. Then, I should reuse the handle to download objects with same domain
name. So, I can reuse DNS result and make use of persistant connections. Thx.

> Sharing data and adding mutex support is being discussed and I welcome
> everyone to contribute to the discussion and join the effort.
>
> > Actually, I am using libcurl to replace libwww. When the network
> > bandwidth is enough, download speed of multi-threaded libcurl is
> > impressive (doubly faster than libwww, even not tuned).
>
> This is really suprising. How can that be?
>
> > But when the network is congested, libcurl is serveral times slower than
> > libwww. I will happy to work on that if someone has any idea.
>
> This is equally surprising and of course of more concern for us.
>
> I don't have any answers to why this happens. Before we continue on this,
> can you please post more detailed and specific, both test results and
> details on how your tests have been performed?

        The program I am developing is a web browser. I just mark the performance
by downloading a page serveral times and taking the average. With other parts
of browser remains the same, downloading a local(HK) page requires 14s for
libcurl version, but 30s for libwww version. However, for www.cnn.com, the
result is just reversed. Libwww seems downloading much faster.
        I am still investigating it. What information do I need to supply in
addition?

>
> Did anyone else notice any symthoms like this?
>
> Then, we should do some careful ethereal'ing or something, to track down
> network packet flows and see if they look weird.
>
> Of course we should also dig into the code and see if we're doing something
> utterly silly in there. I really can't think of what we can possibly do to
> make it not handle congested situations fine. We do the same for all
> conditions.

>
> I trust you've done these tests using libcurl 7.9.5?

        Yes. I am using the newest version.
Received on 2002-03-27