metalink: remove #7176

bagder · 2021-06-02T14:09:14Z

Warning: this will make existing curl command lines that use metalink to
stop working.

Reasons for removal:

We've found several security problems and issues involving the
metalink support in curl. The issues are not detailed here. When
working on those, it become apparent to the team that several of the
problems are due to the system design, metalink library API and what
the metalink RFC says. They are very hard to fix on the curl side
only.
The metalink usage with curl was only very briefly documented and was
not following the "normal" curl usage pattern in several ways, making
it surprising and non-intuitive which could lead to further security
issues.
The metalink library was last updated 6 years ago and wasn't so
active the years before that either. An unmaintained library means
there's a security problem waiting to happen. This is probably reason
enough.
Metalink requires an XML parsing library, which is complex code (even
the smaller alternatives) and to this day often gets security
updates.
Metalink is not a widely used curl feature. In the 2020 curl user
survey, only 1.4% of the responders said that they'd are using
it. Searching the web also show very few traces of it being used,
even with other tools.
The torrent format and associated technology clearly won for
downloading large files from multiple sources in parallel.

Warning: this will make existing curl command lines that use metalink to stop working. Reasons for removal: 1. We've found several security problems and issues involving the metalink support in curl. The issues are not detailed here. When working on those, it become apparent to the team that several of the problems are due to the system design, metalink library API and what the metalink RFC says. They are very hard to fix on the curl side only. 2. The metalink usage with curl was only very briefly documented and was not following the "normal" curl usage pattern in several ways, making it surprising and non-intuitive which could lead to further security issues. 3. The metalink library was last updated 6 years ago and wasn't so active the years before that either. An unmaintained library means there's a security problem waiting to happen. This is probably reason enough. 4. Metalink requires an XML parsing library, which is complex code (even the smaller alternatives) and to this day often gets security updates. 5. Metalink is not a widely used curl feature. In the 2020 curl user survey, only 1.4% of the responders said that they'd are using it. Searching the web also show very few traces of it being used, even with other tools. 6. The torrent format and associated technology clearly won for downloading large files from multiple sources in parallel.

ytrezq · 2021-06-02T22:09:06Z

I disagree on point 6 though, torrent will never support e2dk or gnunet or ipts unlike MetaLink. But I recognize curl only supports servers protocols and not p2p ones.

bagder · 2021-06-03T10:33:47Z

Assuming no major obstacle, I will merge this on June 7th.

kdudka · 2021-06-03T12:00:32Z

I would prefer if --with-libmetalink made the configure script fail rather than being silently ignored. If someone expects metalink to be available, they should learn about its removal rather sooner than later.

bagder · 2021-06-03T12:32:14Z

I would prefer if --with-libmetalink made the configure script fail rather than being silently ignored.

Good idea!

danielgustafsson · 2021-06-03T12:35:18Z

Another idea could be to move metalink support to being an experimental feature (even though experimental generally means moving the other way) requiring an explicit opt-in before removing it. Iff there is an outcry after the release then there is a way for powerusers to keep their systems running while we take the discussion on how to proceed. FWIW, I'm not sure I want this but I wanted to raise it in the discussion before the PR is merged.

bagder · 2021-06-03T12:43:52Z

I don't think we should merge even an experimental feature with known security problems so they would still need to be worked on first.

Power users can just revert this removal to get metalink back with a local patch. If there's a strong enough demand to get it back "for real", I expect developer time to get funded to fix the issues.

danielgustafsson · 2021-06-03T12:53:58Z

On 3 Jun 2021, at 14:44, Daniel Stenberg ***@***.***> wrote: I don't think we should merge even an experimental feature with known security problems so they would still need to be worked on first. Power users can just revert this removal to get metalink back with a local patch. If there's a strong enough demand to get it back "for real", I expect developer time to get funded to fix the issues.

Agreed, I mainly wanted the discussion to have covered all angles given how drastic this measure is.

xquery · 2021-06-03T16:13:53Z

maybe this has already been discussed here or elsewhere - apologies

I have no sense of metalink usage ( I see very few respondents to survey use it) ... adhoc observation is that there is not a lot of adoption ... in the best of worlds (good developer support of exotic feature) my pref would be that said feature would not be part of the core ... considering we have less then good implementation - its a 'no brainer' to deprecate.

Alternately is it time to discuss (prob again) how to formalise pluggability beyond obvious routes .. it makes my brain hurt as its so easy to extend curl/libcurl and not warrant inclusion into core repo ... but is it getting to the point where we need more points of extensibility ?

bagder · 2021-06-03T17:47:53Z

is it getting to the point where we need more points of extensibility ?

Perhaps. I don't think this is the last straw that breaks the camel's back though because I don't think it is popular enough.

Extensibility, plugins and add-ons are mentioned every once in a while. It's just that nobody has yet even tried to write up a suggestion of how it could work - at any level of ambition - so I think we're still not quite there yet. Personally, I'm casually skeptic because I think both curl and libcurl are what they are partly because they provide feature-packed all-in-one packages.

Warning: this will make existing curl command lines that use metalink to stop working. Reasons for removal: 1. We've found several security problems and issues involving the metalink support in curl. The issues are not detailed here. When working on those, it become apparent to the team that several of the problems are due to the system design, metalink library API and what the metalink RFC says. They are very hard to fix on the curl side only. 2. The metalink usage with curl was only very briefly documented and was not following the "normal" curl usage pattern in several ways, making it surprising and non-intuitive which could lead to further security issues. 3. The metalink library was last updated 6 years ago and wasn't so active the years before that either. An unmaintained library means there's a security problem waiting to happen. This is probably reason enough. 4. Metalink requires an XML parsing library, which is complex code (even the smaller alternatives) and to this day often gets security updates. 5. Metalink is not a widely used curl feature. In the 2020 curl user survey, only 1.4% of the responders said that they'd are using it. In 2021 that number was 1.2%. Searching the web also show very few traces of it being used, even with other tools. 6. The torrent format and associated technology clearly won for downloading large files from multiple sources in parallel. Cloes #7176

bagder · 2021-06-07T06:53:38Z

Closed by 265b14d

See upstream pull request curl/curl#7176 which removes metalink support To build with metalink support is no longer available. Closes #78895. Signed-off-by: Carlo Cabrera <30379873+carlocab@users.noreply.github.com> Signed-off-by: BrewTestBot <1589480+BrewTestBot@users.noreply.github.com>

Today curl upstream announced that they are going to completely remove support for metalink from curl already in the next release of curl due to a number of difficult to fix security issues: https://curl.se/mail/archive-2021-06/0006.html curl/curl#7176

Changelog: https://curl.se/changes.html#7_82_0 curl 7.78.0 has not metalink support, see blog post about it [1] and pull request [2]. 1. https://daniel.haxx.se/blog/2021/06/07/bye-bye-metalink-in-curl/ 2. curl/curl#7176 NO_DOC=libcurl submodule bump NO_CHANGELOG=libcurl submodule bump NO_TEST=libcurl submodule bump Closes: tarantool#6029 Closes: tarantool/security#10

Changelog: https://curl.se/changes.html#7_80_0 curl 7.78.0 has not metalink support, see blog post about it [1] and pull request [2]. 1. https://daniel.haxx.se/blog/2021/06/07/bye-bye-metalink-in-curl/ 2. curl/curl#7176 NO_DOC=libcurl submodule bump NO_CHANGELOG=libcurl submodule bump NO_TEST=libcurl submodule bump Closes: tarantool#6029 Closes: tarantool/security#10

Changelog: https://curl.se/changes.html#7_80_0 - curl 7.78.0 has not metalink support, see blog post about it [1] and pull request [2]. - libcurl build requires autotools, added required packages to .travis.mk 1. https://daniel.haxx.se/blog/2021/06/07/bye-bye-metalink-in-curl/ 2. curl/curl#7176 NO_DOC=libcurl submodule bump NO_CHANGELOG=libcurl submodule bump NO_TEST=libcurl submodule bump Closes: tarantool#6029 Closes: tarantool/security#10

Changelog: https://curl.se/changes.html#7_80_0 - curl 7.78.0 has not metalink support, see blog post about it [1] and pull request [2]. - libcurl build requires autotools, added required packages to .travis.mk NOTE: Job release_asan_clang11 in continuous integration triggers memory leak in Curl_resolv(): 022] # Starting group: http_client.sock_family:"AF_UNIX" [022] Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'> [022] BrokenPipeError: [Errno 32] Broken pipe [022] [022] ================================================================= [022] ==1591052==ERROR: LeakSanitizer: detected memory leaks [022] [022] Indirect leak of 86 byte(s) in 1 object(s) allocated from: [022] #0 0x4a6072 in calloc (/home/ubuntu/actions-runner/_work/tarantool/tarantool/src/tarantool+0x4a6072) [022] tarantool#1 0x990839 in Curl_resolv (/home/ubuntu/actions-runner/_work/tarantool/tarantool/src/tarantool+0x990839) [022] [022] Indirect leak of 74 byte(s) in 1 object(s) allocated from: [022] #0 0x4a6072 in calloc (/home/ubuntu/actions-runner/_work/tarantool/tarantool/src/tarantool+0x4a6072) [022] tarantool#1 0x9907c5 in Curl_resolv (/home/ubuntu/actions-runner/_work/tarantool/tarantool/src/tarantool+0x9907c5) [022] [022] SUMMARY: AddressSanitizer: 160 byte(s) leaked in 2 allocation(s). Memory leak reproduced on default HTTP client and couldn't be reproduced with new clients created by require('http.client'):new(). Function Curl_resolv() has been added LSAN suppression list and new issue was submitted, see [3]. 1. https://daniel.haxx.se/blog/2021/06/07/bye-bye-metalink-in-curl/ 2. curl/curl#7176 3. tarantool#7080 NO_DOC=libcurl submodule bump NO_CHANGELOG=libcurl submodule bump NO_TEST=libcurl submodule bump Closes: tarantool#6029 Closes: tarantool/security#10

Changelog: https://curl.se/changes.html#7_80_0 - curl 7.78.0 has not metalink support, see blog post about it [1] and pull request [2]. - libcurl build requires autotools, added required packages to .travis.mk NOTE: Job release_asan_clang11 in continuous integration triggers memory leak in Curl_resolv(): 022] # Starting group: http_client.sock_family:"AF_UNIX" [022] Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'> [022] BrokenPipeError: [Errno 32] Broken pipe [022] [022] ================================================================= [022] ==1591052==ERROR: LeakSanitizer: detected memory leaks [022] [022] Indirect leak of 86 byte(s) in 1 object(s) allocated from: [022] #0 0x4a6072 in calloc (/home/ubuntu/actions-runner/_work/tarantool/tarantool/src/tarantool+0x4a6072) [022] #1 0x990839 in Curl_resolv (/home/ubuntu/actions-runner/_work/tarantool/tarantool/src/tarantool+0x990839) [022] [022] Indirect leak of 74 byte(s) in 1 object(s) allocated from: [022] #0 0x4a6072 in calloc (/home/ubuntu/actions-runner/_work/tarantool/tarantool/src/tarantool+0x4a6072) [022] #1 0x9907c5 in Curl_resolv (/home/ubuntu/actions-runner/_work/tarantool/tarantool/src/tarantool+0x9907c5) [022] [022] SUMMARY: AddressSanitizer: 160 byte(s) leaked in 2 allocation(s). Memory leak reproduced on default HTTP client and couldn't be reproduced with new clients created by require('http.client'):new(). Function Curl_resolv() has been added LSAN suppression list and new issue was submitted, see [3]. 1. https://daniel.haxx.se/blog/2021/06/07/bye-bye-metalink-in-curl/ 2. curl/curl#7176 3. #7080 NO_DOC=libcurl submodule bump NO_CHANGELOG=libcurl submodule bump NO_TEST=libcurl submodule bump Closes: #6029 Closes: https://github.com/tarantool/security/issues/10

bagder added the cmdline tool label Jun 2, 2021

bagder mentioned this pull request Jun 2, 2021

Curl is ignoring content encoding in the case of metalink #7111

Closed

xquery mentioned this pull request Jun 3, 2021

missing Metalink support, please build the docker with it. curl/curl-docker#34

Closed

bagder linked an issue Jun 3, 2021 that may be closed by this pull request

Curl is ignoring content encoding in the case of metalink #7111

Closed

fixup make configure --with-libmetalink fail

7023352

bagder closed this Jun 7, 2021

bagder deleted the bagder/rm-metalink branch June 7, 2021 06:53

ghost mentioned this pull request Jun 7, 2021

curl: remove metalink support Homebrew/homebrew-core#78895

Closed

6 tasks

dheater mentioned this pull request Jul 21, 2021

[request] libcurl/7.78.0 conan-io/conan-center-index#6449

Closed

heitbaum mentioned this pull request Aug 12, 2021

curl: update to 7.78.0 and HSTS LibreELEC/LibreELEC.tv#5120

Merged

Crghilardi mentioned this pull request Sep 2, 2021

Bumping to curl 7.78.0 JuliaPackaging/Yggdrasil#3575

Merged

KoalaBear84 mentioned this pull request Jan 15, 2023

Support RFC5854 metalink file output KoalaBear84/OpenDirectoryDownloader#20

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

metalink: remove #7176

metalink: remove #7176

bagder commented Jun 2, 2021

ytrezq commented Jun 2, 2021

bagder commented Jun 3, 2021

kdudka commented Jun 3, 2021

bagder commented Jun 3, 2021

danielgustafsson commented Jun 3, 2021

bagder commented Jun 3, 2021

danielgustafsson commented Jun 3, 2021 via email

xquery commented Jun 3, 2021

bagder commented Jun 3, 2021 •

edited

bagder commented Jun 7, 2021

metalink: remove #7176

metalink: remove #7176

Conversation

bagder commented Jun 2, 2021

ytrezq commented Jun 2, 2021

bagder commented Jun 3, 2021

kdudka commented Jun 3, 2021

bagder commented Jun 3, 2021

danielgustafsson commented Jun 3, 2021

bagder commented Jun 3, 2021

danielgustafsson commented Jun 3, 2021 via email

xquery commented Jun 3, 2021

bagder commented Jun 3, 2021 • edited

bagder commented Jun 7, 2021

bagder commented Jun 3, 2021 •

edited