Re: Curl's network performance is slower than IDM.

Daniel Stenberg <daniel_at_haxx.se> 于2020年6月17日周三 下午10:58写道：
>
> On Tue, 16 Jun 2020, Hongyi Zhao wrote:
>
> > the IDM latest version is run from within wine [1] git master version
> > compiled by myself on Ubuntu 20.04.
>
> Oh ok, wow.
>
> > the results is the same: IDM beats curl.
>
> Presumably also by a very large margin?

The results are the ones I've posted previously in this email.
Basically, IDM performs a result which is around 1 ~ 3 MB/s higher
than what curl gives.

>
> >> BTW, if curl is slow then surely you could also get faster download speeds
> >> with other transfer tools? Have you identified any such, that perhaps runs
> >> on Linux?
> >
> > To be frank, it seems there is no other tool/library on Linux which can
> > afford so many features as (lib)curl.
>
> Well, neither does IDM. But there are plenty of tools and libraries that can
> download a URL from the Internet and that's the operation we're comparing
> here. wget is a popular one for example. Does wget also beat curl in this
> setup by a large margin?

No, wget and curl give similar results. OTOH, wget has fewer features
than curl, say, socks5 proxy supporting, so, I rarely use it.

>
> >> It's not so much a threading issue as multiple TCP connections
> >
> > Still interesting again. Is there any example for libcurl to establish
> > multiple TCP connections within one threading?
>
> The libcurl multi interface is providing exactly that ability. Which btw you
> can use with the cmdline tool if you do -Z transfers.

When I try with this option, it doesn't give any boosting for speed,
instead, it will cause the downloading speed fluctuating in a wide
range.

>
> > So, here, by saying *connections*, you still refer to TCP connections,
> > instead of the number of thread/process.
>
> Yes. Because what transfer tools usually find that they benfit from is to open
> multiple connections, as doing many connections can often get a larger total
> bandwidth than a single connection does (on the expense of other traffic on
> the network of course) since TCP connections are designed to be "fair" and
> allow others to also use the network. Also, middle boxes are known to exist to
> limit and restrict traffic on a per-connection basis.
>
> The network doesn't know how many threads or process you run in your machine
> so as long as your code manages to read/write the sockets at a sufficiently
> high speed within a single thread the number of threads will have zero effect
> on that.

But, the PyIDM's developer told me the following point of view on the
*TCP connections* discussed here:
https://github.com/pyIDM/PyIDM/issues/107#issuecomment-645273695.

I excerpted the reply as the following:

I don’t think you can open multiple tcp connections asking for the
same chunk range of data
For example curl-multi can run multiple easy handles from one thread
but still every handle will request different chunk range from server
then your application will concatenate these chunks into one file

Otherwise you will get the same data over and over

I believe he meant IDM might use multi-connections silently ignoring
your settings
The only way to make sure it can’t do this is to download a file from
a server which doesn’t support “range requests” which the server
doesn’t have ‘Accept-Ranges: bytes’ in response headers

You can easily identify this by pasting url in PyIDM and it will show
‘Resumable: No’
If i find an example url i will send to you

>
> --
>
> / daniel.haxx.se | Commercial curl support up to 24x7 is available!
> | Private help, bug fixes, support, ports, new features
> | https://www.wolfssl.com/contact/

-- 
Hongyi Zhao <hongyi.zhao_at_gmail.com>
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette:   https://curl.haxx.se/mail/etiquette.html