On Tue, May 15, 2012 at 11:39 AM, Anthony Bryan <anthonybryan_at_gmail.com> wrote:
> On Mon, May 14, 2012 at 12:18 PM,  <curl-users-request_at_cool.haxx.se> wrote:
>> Date: Tue, 15 May 2012 01:18:07 +0900
>> From: Tatsuhiro Tsujikawa <tatsuhiro.t_at_gmail.com>
>> To: the curl tool <curl-users_at_cool.haxx.se>
>> Subject: Re: Metalink support patch for curl
>> Message-ID:
>>        <CAPyZ6=KAgQv1gnp2QOPUUQKUcEm3yuRJmehy0JXbSxY=XTN2HA_at_mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> On Sun, May 13, 2012 at 9:28 PM, Tatsuhiro Tsujikawa
>> <tatsuhiro.t_at_gmail.com> wrote:
>>> On Sat, May 12, 2012 at 12:09 AM, Tatsuhiro Tsujikawa
>>> <tatsuhiro.t_at_gmail.com> wrote:
>>>> On Fri, May 11, 2012 at 5:01 AM, Anthony Bryan <anthonybryan_at_gmail.com> wrote:
>>>>> On Tue, May 8, 2012 at 11:28 AM, ?<curl-users-request_at_cool.haxx.se> wrote:
>>>>>> Message: 1
>>>>>> Date: Wed, 9 May 2012 00:28:30 +0900
>>>>>> From: Tatsuhiro Tsujikawa <tatsuhiro.t_at_gmail.com>
>>>>>> To: the curl tool <curl-users_at_cool.haxx.se>
>>>>>> Subject: Re: Metalink support patch for curl
>>>>>> Message-ID:
>>>>>> ? ? ? ?<CAPyZ6=L1At3YREO_y21VtVgYqwt=bEPECBXWpkZuqu_jTmLZzw_at_mail.gmail.com>
>>>>>> Content-Type: text/plain; charset="iso-8859-1"
>>>>>>
>>>>>> On Mon, May 7, 2012 at 1:18 AM, Tatsuhiro Tsujikawa
>>>>>> <tatsuhiro.t_at_gmail.com> wrote:
>>>>>
>>>>>> I included above change in the attached patch.
>>>>>> I also fixed the issue when content-type has parameters. Now you can download
>>>>>> http://openoffice.mirrorbrain.org/stable/3.3.0/OOo-SDK_3.3.0_Linux_x86-64_install-deb_en-US.tar.gz.metalink
>>>>>
>>>>> thanks, that works for me!
>>>>>
>>>>> could you also sanitize <file name=""> because I noticed I could use
>>>>> <file name="../foo"> or <file name="/root/bar"> and traverse
>>>>> directories.
>>>>>
>>>>> is it possible to have this sanitizing in libmetalink? then it would
>>>>> only need to be done once there for any app that uses it. or maybe it
>>>>> is better suited to these curl patches, I don't know.
>>>>>
>>>>> from http://tools.ietf.org/html/rfc5854#section-4.1.2.1
>>>>>
>>>>> ? ? ?Security Note: The path MUST NOT contain any directory traversal
>>>>> ? ? ?directives or information. ?The path MUST be relative. ?The path
>>>>> ? ? ?MUST NOT begin with a "/", "./", or "../"; contain "/../"; or end
>>>>> ? ? ?with "/..".
>>>>>
>>>>
>>>> I agree to make this sanitizing in libmetalink. Good idea.
>>>>
>>>
>>> Fixed in libmetalink trunk.
>>>
>>
>> I fixed the bug that causes segmentation fault if name attribute of
>> file element in Metalink is invalid (e.g., name starts with "../") and
>> as a result there are nothing download.
>> I also fixed the same bug when the number of resources is 0.
>
> awesome, good catch!
>
>> In addition to the above fixes, I added the code to always create
>> directory hierarchy for Metalink download.
>> The directory hierarchy creation is needed because name attribute of
>> file element in Metalink contains directory information. The next
>> libmetalink release will ensure that name element is relative and must
>> not contain directory traversal directives.
>
> great! thanks for all the libmetalink work and also for mentoring the student!
>
> one possibly last thing, do we want to support transparent usage,
> where metalinks are advertised with a Link header?
> although this is mostly used with .meta4 files, it might be nice to
> have in place for when libmetalink supports .meta4 (which seems like
> it will be soonish) :)
>
> from Daniel:
>> The existing hash code is not exposed to the client code but is only libcurl
>> internal so far. We do have some code that we still re-use in curl from
>> libcurl under the hood that isn't using the API - just in the name of avoiding
>> code duplication - and there might be reasons to do this for some hash
>> functions as well. Although I'm not sure they are that easily shared like
>> this. It needs to be investigated closer.
>
> Daniel, how do you suggest hash checking should be implemented?
> do you want hash checking implemented before merging?
>

I added checksum checking feature in the attached patch set (still
cumulative...).
The behind story of this move follows:
I felt, without checksum checking, Metalink support is somewhat less
attractive, so I invested my time
to check the hash function usage in libcurl.
According to my research, hash functions used are md4 and md5.
They are not directly used from curl (tool) source code.
I looked the code in curl_md5.h and md5.c and found that it has
abstraction layer for several crypto libraries.
I extended this abstraction so that it is not MD5 hash function dependent.
The extended version is in tool_metalink.{h,c}.
I'm used to libgcrypt/nettle/OpenSSL, so currently, my abstractions
support those libraries.
No code added for NSS.

The current implementation supports sha-256, sha-1 and md5. Of course,
the "strongest" hash function is used if there are several functions
available.

Best regards,

Tatsuhiro Tsujikawa

> Tatsuhiro also asked if there were some tests suitable as a template
> to add Metalink test?
>
> & should the patch be cumulative or individual (like they are now)?
>
> to everyone else:
>
> maybe we should have introduced things earlier, but if you haven't
> heard of Metalink, it's a way to provide mirrors, hashes, signatures,
> and other information for download clients.
>
> it's used in a good number of places (KDE, Xfce, LibreOffice,
> OpenOffice.org, Fedora, Ubuntu, openSUSE, sugar, XBMC, FSF, etc)
>
> a good server to test from, you can click on "Details" for any file
> here to see the metalink and other information, or just append
> ".metalink" to any filename.
> http://download.documentfoundation.org/libreoffice/stable/3.5.3/deb/x86/
>
> please help us test these patches!
>
> usage is:
> local metalink: curl --metalink example.metalink
> remote metalink: curl http://example.com/example.metalink -O
>
> --
> (( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
>   )) Easier, More Reliable, Self Healing Downloads
>
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-users
> FAQ:        http://curl.haxx.se/docs/faq.html
> Etiquette:  http://curl.haxx.se/mail/etiquette.html

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html