Re: Curl's network performance is slower than IDM.

On Tue, 16 Jun 2020, Hongyi Zhao wrote:

>> Are these transfer speeds somewhat stable and reliable in repeated tests?
>
> I've conducted several tests at different times against this URL using both
> curl and IDM, though the speeds may vary each time, IDM always beats curl.

The huge difference is what makes this so weird. I mean, I wouldn't be
surprised if a Windows-specific tool maybe can do transfers slightly faster
somehow when tickling the OS the right way, but I can't understand what would
make the difference that big. To me this smells like a setting/config
somewhere that needs to be tweaked or something, rather tham just a basic
software architectural mistake or overhead.

Uh, but you compare against curl on Ubuntu? IDM is a Windows-only tool, right?
You're *sure* there aren't differences between the different system setups
that could explain for (parts of) this? Wouldn't it make more sense to compare
with curl running on the same Windows machine than runs IDM?

>> Do the same symptoms reproduce even if you don't use a SOCKS proxy?
>
> It seems this URL is banned for my location. Without using the socks5 proxy,
> the connection won't be established.

If you try against other test servers/proxies, I presume you see similar
differences as well?

>> From my dev machine in Sweden, my curl reached 30MB/s from this URL.
>
> Have you also tested with IDM for comparison just like I've done?

I don't have a comparable Windows setup to test on. Also, this application is
a commercial one that isn't open source so I rather avoid it for that reason
too.

BTW, if curl is slow then surely you could also get faster download speeds
with other transfer tools? Have you identified any such, that perhaps runs on
Linux?

>> I'm not aware of any. Reading on the IDM site, it seems very content about
>> its use of multiple connections and ranges to download data.
>
> All these techniques can also be done with libcurl-multi which is one
> of the goals of this project: https://github.com/pyIDM/PyIDM

I'm aware. I'm just trying to make sure that we compare apples with apples.

> Interesting. IMO, the curl itself doesn't support multi threading technique,

It's not so much a threading issue as multiple TCP connections that's going to
make a difference.

> and for IDM, I set the max. conn. number = 1 for testings.
>
> But here, you told `a single TCP stream `. I still cannot figure out how to
> check or ensure I'm meeting this condition. Any more hints?

I can only think of using wireshark or something to verify what's going on
under the hood but I guess that we also have no actual reason to suspect that
"conn. number = 1" isn't actually limiting the number of connections to 1.

-- 
  / daniel.haxx.se | Commercial curl support up to 24x7 is available!
                   | Private help, bug fixes, support, ports, new features
                   | https://www.wolfssl.com/contact/
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html