On Mon, 7 Apr 2008, Dr. Peter Poeml wrote:

> You are probably aware of metalinks (http://metalinker.org), and how they
> can be used to make downloas from HTTP, FTP and other sources (that curl
> handles) better for users, by implementing failover and integrity checking.

Yes, and we even provide curl downloads as metalink files!

> In fact, it is an enthralling thought to me that libcurl could understand
> and handle metalinks already, because all what I proposed in [2] would come
> for free. (Well, free for libzypp that is. :-) But the interesting thing is
> that lots of software uses libcurl, so it would all benefit from such a
> capability.
>
> For openSUSE, it would be so very beneficial that there is actually a GSoC
> project about implementing the proposal from [2]. With metalink support in
> libcurl it could be rather something which is generally useful, though.

I've had this discussion at length with Anthony Bryan privately in the past
and I've bounced back a lot of feedback on the actual xml format to him and I
believe some of that were taken into account and changed the format. Of course
this was before it "settled" and started to get adopted.

I think metalink is a great idea and the file format is (the last time I
checked it out, I can't seem to find the docs now) mostly making sense.

However, and here it comes, I have little to no understanding of the idea that
libcurl would add support for this natively. metalink is just an XML format
that sets up resources for an application to where and how it can download
files, and libcurl does indeed support most of the protocols that such URLs
can use. libcurl is a data transfer library that is oriented around a given
URL and the URL in question has a 1:1 relationship to what protocol it is and
it is always content-agnostic.

metalink is application layer, not transport. Adding metalink to libcurl would
mean that all of a sudden libcurl would transfer a file and actually parse the
(XML!) contents of that file and then get (possibly) multiple streams using
multiple protocols based on what that parsing gave. It is just so many new
things and violations against key libcurl concepts that I cannot see this
done.

Metalink isn't even a standard so we would then more or less open the gates
for further random efforts to introduce similar ideas and whatnot and where
would we draw the line? Currently I think we have a pretty solid border drawn
in the sand and we don't cross that line (on purpose).

And frankly, there is only one and one reason only (mentioned and that I can
think of) for libcurl to support this feature and I that is because libcurl is
already widely adopted it would be easier for metalink to conquer the world by
sneaking in the backdoor with libcurl as then a large amount of applications
would support it with no additional efforts at all. But sorry, I don't think
that's a good enough reason to break or change these important key
concepts/limits of libcurl. (Actually, I think it is a bit foolish to think
that adding metalink to libcurl would make all these applications
automatically support metalink as there would be several arguments against
that.)

As I've said before, I think one of our biggest challenges in this project is
to limit what libcurl does, to not allow it to grow in all directions, to keep
the scope and to maintain focus.

A metalink file transfer library could be made as a layer on top of libcurl,
and I think that is the only logical and sensible way.

Addinng metalink support to the curl tool however, seems like a very good idea
to me...

-- 
  Commercial curl and libcurl Technical Support: http://haxx.se/curl.html