> From: Daniel Stenberg <daniel_at_haxx.se>
> Subject: Re: metalink support
> To: the curl tool <curl-users_at_cool.haxx.se>
> Message-ID: <Pine.LNX.4.64.0804151010570.28700_at_yvahk3.pbagnpgbe.fr>
> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
>
> On Tue, 15 Apr 2008, Anthony Bryan wrote:
>
> > Tatsuhiro Tsujikawa is writing a small metalink parser in C, to be used by
> > wget.
> >
> > Could this be used for curl as well?
>
> Absolutely, there's no practical reason why we couldn't! It would of course
> require that the source code is donated to us using a suitable license (which
> wouldn't be the same license wget wants it as).

I was wondering about that too. Is a dual license acceptable/possible?

> Of course it begs the question if that ends up as a complete speparate library
> both curl and wget can use?
>
> > Micah Cowan, wget's maintainer said "Of course, curl and Wget would also
> > require interfacing code to _use_ the parser, as well as to fall back to the
> > alternatives when one of them fails."
>
> Indeed.
>
> The parsing of metalink is actually the lesser part of the work I'd say
> (although of course not insignificant), as there's no real internal support in
> curl for multiple simultaneous streams although libcurl offers quite extensive
> support for doing it.
>
> In wget that lacking infrastructure is probably an even bigger hurdle since
> they don't even have an underlying lib that supports this.
>
> Changing curl's internals to support parallell transfers is something I've
> been wanting to do for ages though, so it would be a nice additional bonus
> if/when metalink support is added. Of course, I've also been a bit "scared" to
> introduce parallell transfers in general since it opens up a whole new can of
> worms on how things should work and how to display the transfers and how to
> display failures and...

Parallel transfers would be great, but would you be opposed to
starting off with something simpler?

I should have posted this as well, but this is what wget is looking for:

Download from a single URL (no multi-source downloads).
curl would get a URL from the mirror list in the .metalink (preferably
one with the highest priority) and verify the whole file checksum.
Optionally but preferably, (1) if that mirror went down curl could
switch to the next highest priority URL and (2) if there was an error
in transfer it could request that chunk again and compare against the
chunk checksums.

Also, another favor. :) Could you update
http://cool.haxx.se/cvs.cgi/curl-www/metalink.cgi which creates
metalinks on the download page? Basically, right now it's <hash
type="pgp"> and it should be <signature type="pgp">. This is something
you suggested a while back, I changed things but there must have been
some miscommunication. There's now a metalink client that verifies
signatures, and I think curl is the only place that has them in their
.metalinks!

    if( -r "download/$archive.asc" ) {
        print "<signature type=\"pgp\" file=\"$archive.asc\">\n";
        open(SIG, "<download/$archive.asc");
        while(<SIG>) {
           print CGI::escapeHTML($_);
            }
        close(SIG);
            print "</signature>\n";

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
 )) Easier, More Reliable, Self Healing Downloads