Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Honor 429 HTTP Code / Retry-After header #3794

Closed
john-hascall opened this issue Apr 19, 2019 · 9 comments
Closed

Honor 429 HTTP Code / Retry-After header #3794

john-hascall opened this issue Apr 19, 2019 · 9 comments

Comments

@john-hascall
Copy link

The command line client's --retry option should recognize and honor the 429 HTTP status return and the accompanying 'Retry-After' header.

Affects all versions, but I happen to be running:

curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.28.4 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

on:

Linux 3.10.0-514.26.2.el7.x86_64 #1 SMP Fri Jun 30 05:26:04 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

@bagder
Copy link
Member

bagder commented Apr 19, 2019

I reasonable suggestion methinks, even if that's a very rarely used response code and header...

@john-hascall
Copy link
Author

john-hascall commented Apr 19, 2019 via email

@bagder
Copy link
Member

bagder commented May 3, 2019

I want to address this issue after #3804 has gone in, just to save me from having to do double work.

@mark-kubacki
Copy link
Contributor

At least one server written in Golang does sent this. It's used when it runs out of file descriptors.

Another sensible use is in anything that runs in the cloud, hits a resource limit (clouds are made of rigid boxes), and cold-start times are known.

@bagder
Copy link
Member

bagder commented Aug 6, 2019

@john-hascall
Copy link
Author

john-hascall commented Aug 6, 2019 via email

bagder added a commit that referenced this issue Aug 6, 2019
This is only the libcurl part that provides the information. There's no
user of the parsed value. This change includes three new tests for the
parser.

Ref: #3794
bagder added a commit that referenced this issue Aug 6, 2019
This is only the libcurl part that provides the information. There's no
user of the parsed value. This change includes three new tests for the
parser.

Ref: #3794
bagder added a commit that referenced this issue Aug 6, 2019
This is only the libcurl part that provides the information. There's no
user of the parsed value. This change includes three new tests for the
parser.

Ref: #3794
bagder added a commit that referenced this issue Aug 6, 2019
This is only the libcurl part that provides the information. There's no
user of the parsed value. This change includes three new tests for the
parser.

Ref: #3794
bagder added a commit that referenced this issue Aug 6, 2019
If a Retry-After: header was used in the response, that value overrides
other retry timing options.

Closes #3794
bagder added a commit that referenced this issue Aug 7, 2019
This is only the libcurl part that provides the information. There's no
user of the parsed value. This change includes three new tests for the
parser.

Ref: #3794
@bagder bagder closed this as completed in 640b973 Aug 7, 2019
@john-hascall
Copy link
Author

john-hascall commented Aug 8, 2019 via email

@stain
Copy link
Contributor

stain commented Oct 2, 2019

Thanks @bagder - but when I now try to use this in 7.66.0 against for instance Zenodo API it seems curl is still fails early on 429 HTTP status code - is the retry-logic triggered for 429 just like the 5xx status codes?

@stain
Copy link
Contributor

stain commented Oct 2, 2019

Try for instance

    curl \
    --show-error \
    --fail-early \
    --retry \
    10 \
    --retry-connrefused \
    --location \
    --remote-name-all \
    --dump-header \
    - \
    --header \
    'Accept: application/ld+json' \
    https://zenodo.org/api/records/1299320 \
    https://zenodo.org/api/records/1299783 \
    https://zenodo.org/api/records/1484496 \
    https://zenodo.org/api/records/1308997 \
    https://zenodo.org/api/records/1309962 \
    https://zenodo.org/api/records/1484341 \
    https://zenodo.org/api/records/1310621 \
    https://zenodo.org/api/records/1442453 \
    https://zenodo.org/api/records/1484403 \
    https://zenodo.org/api/records/1421066 \
    https://zenodo.org/api/records/1312623 \
    https://zenodo.org/api/records/1311933 \
    https://zenodo.org/api/records/3343031 \
    https://zenodo.org/api/records/3334688 \
    https://zenodo.org/api/records/3380592 \
    https://zenodo.org/api/records/1484442 \
    https://zenodo.org/api/records/1314672 \
    https://zenodo.org/api/records/3458522 \
    https://zenodo.org/api/records/3445531 \
    https://zenodo.org/api/records/3357806 \
    https://zenodo.org/api/records/3362412 \
    https://zenodo.org/api/records/3458560 \
    https://zenodo.org/api/records/3268809 \
    https://zenodo.org/api/records/1313218 \
    https://zenodo.org/api/records/1312652 \
    https://zenodo.org/api/records/1472385 \
    https://zenodo.org/api/records/1323831 \
    https://zenodo.org/api/records/3336124 \
    https://zenodo.org/api/records/3336148 \
    https://zenodo.org/api/records/1208478 \
    https://zenodo.org/api/records/3338176 \
    https://zenodo.org/api/records/1410052 \
    https://zenodo.org/api/records/3384296 \
    https://zenodo.org/api/records/3270620 \
    https://zenodo.org/api/records/1484480 \
    https://zenodo.org/api/records/3196309 \
    https://zenodo.org/api/records/1484502 \
    https://zenodo.org/api/records/1310034 \
    https://zenodo.org/api/records/1410238 \
    https://zenodo.org/api/records/1408170 \
    https://zenodo.org/api/records/1484424 \
    https://zenodo.org/api/records/1326284 \
    https://zenodo.org/api/records/1473154 \
    https://zenodo.org/api/records/1312685 \
    https://zenodo.org/api/records/1420531 \
    https://zenodo.org/api/records/3459770 \
    https://zenodo.org/api/records/3382423 \
    https://zenodo.org/api/records/1215611 \
    https://zenodo.org/api/records/1311612 \
    https://zenodo.org/api/records/3382259 

stain added a commit to stain/curl that referenced this issue Oct 4, 2019
This completes curl#3794

Also make sure the new tests from curl#4195 are enabled
bagder pushed a commit that referenced this issue Oct 5, 2019
This completes #3794.

Also make sure the new tests from #4195 are enabled

Closes #4465
@lock lock bot locked as resolved and limited conversation to collaborators Jan 1, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Development

No branches or pull requests

4 participants