Re: Curl's network performance is slower than IDM.

Daniel Stenberg <daniel_at_haxx.se> 于2020年6月16日周二 下午10:50写道：
>
> On Tue, 16 Jun 2020, Hongyi Zhao wrote:
>
> >> Are these transfer speeds somewhat stable and reliable in repeated tests?
> >
> > I've conducted several tests at different times against this URL using both
> > curl and IDM, though the speeds may vary each time, IDM always beats curl.
>
> The huge difference is what makes this so weird. I mean, I wouldn't be
> surprised if a Windows-specific tool maybe can do transfers slightly faster
> somehow when tickling the OS the right way, but I can't understand what would
> make the difference that big. To me this smells like a setting/config
> somewhere that needs to be tweaked or something, rather tham just a basic
> software architectural mistake or overhead.
>
> Uh, but you compare against curl on Ubuntu?

Maybe I should have told the testing environments more clearly.
Anyway, I describe again on the testing environments in a more
detailed way as follows:

All testings are done on Ubuntu 20.04.
Curl is installed from the distro's repo.
I'm a registered user of IDM, and the IDM latest version is run from
within wine [1] git master version compiled by myself on Ubuntu 20.04.

[1] Wine source code is cloned from here: https://source.winehq.org/git/wine.git

> IDM is a Windows-only tool, right?

Yes, but I run it from within wine on Ubuntu 20.04, as told above.

> You're *sure* there aren't differences between the different system setups
> that could explain for (parts of) this? Wouldn't it make more sense to compare
> with curl running on the same Windows machine than runs IDM?

I don't have machine which running Microsoft Windows, so I cannot do
the testings on Windows machine.

>
> >> Do the same symptoms reproduce even if you don't use a SOCKS proxy?
> >
> > It seems this URL is banned for my location. Without using the socks5 proxy,
> > the connection won't be established.
>
> If you try against other test servers/proxies, I presume you see similar
> differences as well?

I've tried with this URL
(https://mirrors.tsinghua.edu.cn/ctan/systems/texlive/Images/texlive2020-20200406.iso)
which don't be banned for my location without using any proxy, and the
results is the same: IDM beats curl.

>
> >> From my dev machine in Sweden, my curl reached 30MB/s from this URL.
> >
> > Have you also tested with IDM for comparison just like I've done?
>
> I don't have a comparable Windows setup to test on. Also, this application is
> a commercial one that isn't open source so I rather avoid it for that reason
> too.
>
> BTW, if curl is slow then surely you could also get faster download speeds
> with other transfer tools? Have you identified any such, that perhaps runs on
> Linux?

To be frank, it seems there is no other tool/library on Linux which
can afford so many features as (lib)curl. I really feel so difficult
for me to find an alternative one which support multi threading (in
library mode), http(s)/socks4/5 proxies supporting, and so on.

>
> >> I'm not aware of any. Reading on the IDM site, it seems very content about
> >> its use of multiple connections and ranges to download data.
> >
> > All these techniques can also be done with libcurl-multi which is one
> > of the goals of this project: https://github.com/pyIDM/PyIDM
>
> I'm aware. I'm just trying to make sure that we compare apples with apples.
>
> > Interesting. IMO, the curl itself doesn't support multi threading technique,
>
> It's not so much a threading issue as multiple TCP connections

Still interesting again. Is there any example for libcurl to establish
multiple TCP connections within one threading?

> that's going to make a difference.

>
> > and for IDM, I set the max. conn. number = 1 for testings.
> >
> > But here, you told `a single TCP stream `. I still cannot figure out how to
> > check or ensure I'm meeting this condition. Any more hints?
>
> I can only think of using wireshark or something to verify what's going on
> under the hood but I guess that we also have no actual reason to suspect that
> "conn. number = 1" isn't actually limiting the number of connections to 1.

So, here, by saying *connections*, you still refer to TCP connections,
instead of the number of thread/process.

Regards,

-- 
Hongyi Zhao <hongyi.zhao_at_gmail.com>
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette:   https://curl.haxx.se/mail/etiquette.html