Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--parallel combined with --retry hangs indefinitely #5905

Closed
htasta opened this issue Sep 1, 2020 · 4 comments
Closed

--parallel combined with --retry hangs indefinitely #5905

htasta opened this issue Sep 1, 2020 · 4 comments
Assignees

Comments

@htasta
Copy link

htasta commented Sep 1, 2020

I did this

curl -K list_of_urls

list_of_urls:

--compressed
--parallel
--parallel-max 5
--parallel-immediate
--retry 5
--retry-delay 1
-H "some: header"

-o "out/1"
url=https://url.with.occasional/error.429
-o "out/2"
url=https://url.with.occasional/error.429
-o "out/3"
url=https://url.with.occasional/error.429
...
-o "out/100"
url=https://url.with.occasional/error.429

I tried without --parallel-immediate and without --compressed and setting --retry before each url, same result.
I also tried with --fail (learned from last issue 😉)
If I remove --retry it doesn't hang but well the ones who failed failed 😄

I expected the following

Receive files with content after trying again if it failed with 429 (or any of the other HTTP errors)

What happens

curl hangs and the progress meter counts up but nothing happens, the failed files stay at 0 bytes.
sometimes 15 files are created sometimes 12 sometimes ... can't see a pattern there.

curl/libcurl version

current master branch

[curl -V output]
curl 7.73.0-DEV (x86_64-pc-linux-gnu) libcurl/7.73.0-DEV OpenSSL/1.1.1f zlib/1.2.11
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL TLS-SRP UnixSockets

fails on 7.68.0 in the same/similar way

operating system

Linux xxxYYYzzz 5.7.12-050712-generic SMP x86_64 x86_64 x86_64 GNU/Linux

same result with kernel 5.4.0

@htasta
Copy link
Author

htasta commented Sep 5, 2020

Same issue on Windows 10 using 7.72 x64

@bagder
Copy link
Member

bagder commented Sep 5, 2020

--retry is clearly broken when doing parallel transfers. It needs some careful rethinking and fixing.

bagder added a commit that referenced this issue Sep 5, 2020
@bagder bagder linked a pull request Sep 5, 2020 that will close this issue
@bagder bagder self-assigned this Sep 5, 2020
@htasta
Copy link
Author

htasta commented Sep 5, 2020

Seems to be working now with those fixes.

@bagder bagder closed this as completed in e4e725f Sep 5, 2020
@htasta
Copy link
Author

htasta commented Sep 5, 2020

thx for the quick fix 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging a pull request may close this issue.

2 participants