cURL / Mailing Lists / curl-users / Single Mail

curl-users

Re: multi-threaded/segmented downloads w/ cURL

From: anthony l. bryan <albryan_at_comcast.net>
Date: Mon, 22 May 2006 11:52:20 -0400

> From: Daniel Stenberg <daniel_at_haxx.se>
> On Sun, 7 May 2006, anthony l. bryan wrote:
>
> > (There would usually be a bunch of ftp & http sources
> listed. There's also
> > language/OS information in files, so a .metalink for
> OpenOffice.org lists
> > all the language & Oses, & the client would download the
> file for your
> > language/OS if available).
>
> Whoa, I liked this concept all the way to here. Now all of a
> sudden the
> download tool needs to take decisions and yay, it got even
> harder and more
> complex. IHMO, the system should've stayed at providing a
> list of identical
> files and had the normal web pages point out different ones
> for different
> platforms.
>
> I can easily come up with several situations where the tool
> itself cannot know
> what operating system/platform I want to download for and
> voila, you've
> introduced a new set of options for a download tool like this
> where you need
> to be able to set desired operating system (and language and
> whatever else you
> can make up that differ between different the files within
> the same XML blob).

I do think it could be nice in some situations but, I'll take your guidance
and just go with the simpler list of identical files. That will be easier
too. Thanks, I appreciate it!
 
> > Programs that don't support p2p would just use the multiple
> ftp/http URLs
> > listed, get segments from each in parallel, then run md5sum
> or sha1sum
> > against the finished download & compare checksums. You can
> see a video
> > capture of it working at
> http://www.metalinker.org/implementation.html
>
> The actual downloading of files should be very easily done
> since libcurl
> already offers simultaneous transfers of several files. It
> would just need
> some careful handling of the chunks.
>
> Parsing XML should be easily done too as I hear there are
> excellent libs
> already made for this.
>
> In fact, I'm surprised no one did this as a first proof of
> concept for this
> idea, since it feels like a hack you can pull together in
> relatively little
> time.

Thanks again for help in the general direction. It looks like I've found
someone to work on it (but if anyone else wants to help out, let me know).
Hopefully, it won't take too long to hack something together.

Anthony
Received on 2006-05-22