Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
curl ignores the Retry-After header with fail-on-error option
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Cherish98 via curl-users <curl-users_at_cool.haxx.se>
Date: Thu, 31 Dec 2020 13:41:17 +0000
Hello,
I was trying to download lots of files with curl. Due to rate limit on the server side, I enabled the --retry option so that curl retries when the server returns HTTP 429 Too Many Requests. I also used the fail-on-error (-f, --fail) option to make sure that HTTP error pages are not saved to disk, just in case the last retry still got HTTP 429 or other errors. I think that makes a reasonable use case.
I was expecting curl to use the Retry-After value in the HTTP response for the retry delay period. However, curl (libcurl to be precise, but this issue is more concerned with the tool, as library users can deal with errors programmatically) ignores (almost) all the HTTP headers with the CURLOPT_FAILONERROR option. I would suggest that curl looks for the Retry-After header in case of HTTP error 429, even the request itself is marked as failed.
By the way, on the curl(1) man page, it reads that "Transient error means either: [...] an HTTP 408 or 5xx response code." HTTP 429 was left out here.
Thanks!
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2020-12-31
Date: Thu, 31 Dec 2020 13:41:17 +0000
Hello,
I was trying to download lots of files with curl. Due to rate limit on the server side, I enabled the --retry option so that curl retries when the server returns HTTP 429 Too Many Requests. I also used the fail-on-error (-f, --fail) option to make sure that HTTP error pages are not saved to disk, just in case the last retry still got HTTP 429 or other errors. I think that makes a reasonable use case.
I was expecting curl to use the Retry-After value in the HTTP response for the retry delay period. However, curl (libcurl to be precise, but this issue is more concerned with the tool, as library users can deal with errors programmatically) ignores (almost) all the HTTP headers with the CURLOPT_FAILONERROR option. I would suggest that curl looks for the Retry-After header in case of HTTP error 429, even the request itself is marked as failed.
By the way, on the curl(1) man page, it reads that "Transient error means either: [...] an HTTP 408 or 5xx response code." HTTP 429 was left out here.
Thanks!
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2020-12-31