cURL / Mailing Lists / curl-users / Single Mail

curl-users

Re: multi-threaded/segmented downloads w/ cURL

From: Daniel Stenberg <daniel_at_haxx.se>
Date: Mon, 8 May 2006 14:31:14 +0200 (CEST)

On Sun, 7 May 2006, anthony l. bryan wrote:

> (There would usually be a bunch of ftp & http sources listed. There's also
> language/OS information in files, so a .metalink for OpenOffice.org lists
> all the language & Oses, & the client would download the file for your
> language/OS if available).

Whoa, I liked this concept all the way to here. Now all of a sudden the
download tool needs to take decisions and yay, it got even harder and more
complex. IHMO, the system should've stayed at providing a list of identical
files and had the normal web pages point out different ones for different
platforms.

I can easily come up with several situations where the tool itself cannot know
what operating system/platform I want to download for and voila, you've
introduced a new set of options for a download tool like this where you need
to be able to set desired operating system (and language and whatever else you
can make up that differ between different the files within the same XML blob).

> Programs that don't support p2p would just use the multiple ftp/http URLs
> listed, get segments from each in parallel, then run md5sum or sha1sum
> against the finished download & compare checksums. You can see a video
> capture of it working at http://www.metalinker.org/implementation.html

The actual downloading of files should be very easily done since libcurl
already offers simultaneous transfers of several files. It would just need
some careful handling of the chunks.

Parsing XML should be easily done too as I hear there are excellent libs
already made for this.

In fact, I'm surprised no one did this as a first proof of concept for this
idea, since it feels like a hack you can pull together in relatively little
time.

-- 
  Commercial curl and libcurl Technical Support: http://haxx.se/curl.html
Received on 2006-05-08