Re: Let libcurl return the full list of mirror's URLs for a downloading file.

Daniel Stenberg <daniel_at_haxx.se> 于2020年6月22日周一 上午1:29写道：
>
> On Sun, 21 Jun 2020, Hongyi Zhao via curl-library wrote:
>
> > As you can see, it seems libcurl returns the random URL for the target file.
>
> I would rather say it like this:
>
> The *web server* returns a random URL for the target file.
>
> > I want to know whether libcurl has the capability of obtaining all the
> > potential mirror's URLs.
>
> libcurl is not a magician, it just delivers to the client what the server
> responds. In this case it seems the server does some round robin or something
> and delivers different Location: URLs for different requests.
>
> libcurl doesn't know anything about where the target URL may exist, only the
> server knows that and it apparently delivers one of them for each request.
>

OK. But I find that these random URLs have very different network
performance for downloading the target file. See the following for
more information:

$ curl -x socks5://127.0.0.1:18887
https://mirrors.tuna.tsinghua.edu.cn/ctan/systems/texlive/Images/texlive2020-20200406.iso
> /dev/null
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  5 3813M 5 199M 0 0 10.7M 0 0:05:55 0:00:18 0:05:37 13.4M^C
$ curl -x socks5://127.0.0.1:18887
http://ctan.mirror.colo-serv.net/systems/texlive/Images/texlive2020-20200406.iso
> /dev/null
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 3813M 0 32.3M 0 0 1120k 0 0:58:04 0:00:29 0:57:35 1173k^C

So, try to pick out the high-speed mirror URL is crucial for performance.

I want to do the following jobs based on libcurl's multi-threading capability:

1. During the whole downloading process, let different workers/threads
try to re-establish the connection to the initial URL and get the
redirected target URL. So that libcurl can find as many as possible
mirror URLs.

2. Based on the speed of the different target URL, kill the low-speed
workers/threads and reuse the URL which presents high-speed.

Any hints for doing the trick with libcurl?

-- 
Hongyi Zhao <hongyi.zhao_at_gmail.com>
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html