Curl does not retry on 504 #8845

jakub-bochenski · 2022-05-13T16:00:13Z

I did this

curl --fail-with-body --show-error -vvv --oauth2-bearer SNIP  --retry 5 -X PUT -H 'Content-Type: application/json' -d @- https://graph.microsoft.com/beta/trustFramework/keySets/B2C_1A_PrimaryUserstoreSecret
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 20.190.159.29:443...
* Connected to graph.microsoft.com (20.190.159.29) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [85 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3753 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=graph.microsoft.com
*  start date: Apr  8 23:02:02 2022 GMT
*  expire date: Apr  3 23:02:02 2023 GMT
*  subjectAltName: host "graph.microsoft.com" matched cert's "graph.microsoft.com"
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure TLS Issuing CA 01
*  SSL certificate verify ok.
* Server auth using Bearer with user ''
} [5 bytes data]
> PUT /beta/trustFramework/keySets/B2C_1A_PrimaryUserstoreSecret HTTP/1.1
> Host: graph.microsoft.com
> Authorization: Bearer SNIP
> User-Agent: curl/7.81.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 70
> 
} [70 bytes data]
100    70    0     0  100    70      0     54  0:00:01  0:00:01 --:--:--    54
100    70    0     0  100    70      0     30  0:00:02  0:00:02 --:--:--    30
100    70    0     0  100    70      0     21  0:00:03  0:00:03 --:--:--    21
100    70    0     0  100    70      0     16  0:00:04  0:00:04 --:--:--    16
100    70    0     0  100    70      0     13  0:00:05  0:00:05 --:--:--    13
100    70    0     0  100    70      0     11  0:00:06  0:00:06 --:--:--     0
100    70    0     0  100    70      0      9  0:00:07  0:00:07 --:--:--     0
100    70    0     0  100    70      0      8  0:00:08  0:00:08 --:--:--     0
100    70    0     0  100    70      0      7  0:00:10  0:00:09  0:00:01     0
100    70    0     0  100    70      0      6  0:00:11  0:00:10  0:00:01     0
100    70    0     0  100    70      0      6  0:00:11  0:00:11 --:--:--     0
100    70    0     0  100    70      0      5  0:00:14  0:00:12  0:00:02     0
100    70    0     0  100    70      0      5  0:00:14  0:00:13  0:00:01     0
100    70    0     0  100    70      0      4  0:00:17  0:00:14  0:00:03     0
100    70    0     0  100    70      0      4  0:00:17  0:00:15  0:00:02     0
100    70    0     0  100    70      0      4  0:00:17  0:00:16  0:00:01     0
100    70    0     0  100    70      0      4  0:00:17  0:00:17 --:--:--     0
100    70    0     0  100    70      0      3  0:00:23  0:00:18  0:00:05     0
100    70    0     0  100    70      0      3  0:00:23  0:00:19  0:00:04     0
100    70    0     0  100    70      0      3  0:00:23  0:00:20  0:00:03     0
100    70    0     0  100    70      0      3  0:00:23  0:00:21  0:00:02     0
100    70    0     0  100    70      0      3  0:00:23  0:00:22  0:00:01     0
100    70    0     0  100    70      0      3  0:00:23  0:00:23 --:--:--     0
100    70    0     0  100    70      0      2  0:00:35  0:00:24  0:00:11     0
100    70    0     0  100    70      0      2  0:00:35  0:00:25  0:00:10     0
100    70    0     0  100    70      0      2  0:00:35  0:00:26  0:00:09     0
100    70    0     0  100    70      0      2  0:00:35  0:00:27  0:00:08     0
100    70    0     0  100    70      0      2  0:00:35  0:00:28  0:00:07     0
100    70    0     0  100    70      0      2  0:00:35  0:00:29  0:00:06     0
* Mark bundle as not supporting multiuse
< HTTP/1.1 504 Gateway Timeout
< Transfer-Encoding: chunked
< Content-Type: application/json
< Strict-Transport-Security: max-age=31536000
< request-id: 08882158-a60d-4656-a5d1-322449658e3e
< client-request-id: 08882158-a60d-4656-a5d1-322449658e3e
< x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"North Europe","Slice":"E","Ring":"4","ScaleUnit":"000","RoleInstance":"DU6PEPF00002DA8"}}
< Date: Fri, 13 May 2022 15:42:20 GMT
< 
{ [207 bytes data]
100   271    0   201  100    70      6      2  0:00:35  0:00:30  0:00:05    52
* Connection #0 to host graph.microsoft.com left intact
curl: (22) The requested URL returned error: 504

I expected the following

Retry 5 times, then fail with code 22 if it still not suceeded

curl/libcurl version

curl/7.81.0

operating system

/ # cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.15.0_alpha20210804
PRETTY_NAME="Alpine Linux edge"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
``

The text was updated successfully, but these errors were encountered:

dfandrich · 2022-05-13T17:01:07Z

On Fri, May 13, 2022 at 09:00:31AM -0700, jakub-bochenski wrote: I did this curl --fail-with-body --show-error -vvv --oauth2-bearer SNIP --retry 5 -X PUT -H 'Content-Type: application/json' -d @- https://graph.microsoft.com/beta/trustFramework/keySets/B2C_1A_PrimaryUserstoreSecret

...

curl: (22) The requested URL returned error: 504 I expected the following Retry 5 times, then fail with code 22 if it still not suceeded

This seems to be related to the use of --fail-with-body. Dropping that option makes curl retry as expected. --fail-with-body is documented to "Return an error on server errors where the HTTP response code is 400 or greater" so it could be argued this is correct behaviour. Except, the documentation compares this option to --fail which *does* do the retries before returning an error, so I suspect this is actually unintentional curl behaviour. Dan

jakub-bochenski · 2022-05-13T17:02:54Z

Well uh, how else can I get URL to return a 22 code on error, print the body and still do retries?

dfandrich · 2022-05-13T17:20:55Z

Well, you could continue arguing here that it's a bug that somebody needs fix. Or, you could push the proposal that JSON output should include the body. Or, you could drop the --fail-with-body option and parse the output yourself to extract the last body. Or, there are probably other ways neither of us have thought of yet.

jakub-bochenski · 2022-05-13T17:24:12Z

Or, you could push the proposal that JSON output should include the body.

What do you mean? What is "JSON output" in context of curl?

dfandrich · 2022-05-13T17:32:01Z

https://daniel.haxx.se/blog/2020/03/17/curl-write-out-json/ Somebody subsequently proposed that the request body be included as a JSON key, but this was rejected (with good reason, I should add). I'm just suggesting that this proposal, if the objections could be overcome, might be another way to solve your problem if it's decided that --fail-with-body is doing the right thing now and won't be changed, which I personally think is not the case.

jakub-bochenski · 2022-05-13T17:35:05Z

Thanks. FTR I'm after response not request

dfandrich · 2022-05-13T17:55:13Z

%{json} gives you a bit of both.

bagder · 2022-05-13T21:46:47Z

It might be as easy as this:

diff --git a/src/tool_operate.c b/src/tool_operate.c
index daaf0bcd4..67de59c0a 100644
--- a/src/tool_operate.c
+++ b/src/tool_operate.c
@@ -433,11 +433,11 @@ static CURLcode post_per_transfer(struct GlobalConfig *global,
       curl_easy_getinfo(curl, CURLINFO_OS_ERRNO, &oserrno);
       if(ECONNREFUSED == oserrno)
         retry = RETRY_CONNREFUSED;
     }
     else if((CURLE_OK == result) ||
-            (config->failonerror &&
+            ((config->failonerror || config->failwithbody) &&
              (CURLE_HTTP_RETURNED_ERROR == result))) {
       /* If it returned OK. _or_ failonerror was enabled and it
          returned due to such an error, check for HTTP transient
          errors to retry on. */
       long protocol = 0;

... in the same way --fail already does. Reported-by: Jakub Bochenski Fixes #8845

bagder added HTTP cmdline tool labels May 13, 2022

bagder added a commit that referenced this issue May 13, 2022

tool_operate: make sure --fail-with-body works with --retry

e055e37

... in the same way --fail already does. Reported-by: Jakub Bochenski Fixes #8845

bagder linked a pull request May 13, 2022 that will close this issue

make sure --fail-with-body works with --retry #8847

Closed

bagder closed this as completed in 0356804 May 16, 2022

bagder mentioned this issue Jul 1, 2022

--retry does not retry when combined with --fail-with-body #9078

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Curl does not retry on 504 #8845

Curl does not retry on 504 #8845

jakub-bochenski commented May 13, 2022 •

edited

dfandrich commented May 13, 2022 via email

jakub-bochenski commented May 13, 2022 •

edited

dfandrich commented May 13, 2022 via email

jakub-bochenski commented May 13, 2022

dfandrich commented May 13, 2022 via email

jakub-bochenski commented May 13, 2022

dfandrich commented May 13, 2022 via email

bagder commented May 13, 2022

Curl does not retry on 504 #8845

Curl does not retry on 504 #8845

Comments

jakub-bochenski commented May 13, 2022 • edited

I did this

I expected the following

curl/libcurl version

operating system

dfandrich commented May 13, 2022 via email

jakub-bochenski commented May 13, 2022 • edited

dfandrich commented May 13, 2022 via email

jakub-bochenski commented May 13, 2022

dfandrich commented May 13, 2022 via email

jakub-bochenski commented May 13, 2022

dfandrich commented May 13, 2022 via email

bagder commented May 13, 2022

jakub-bochenski commented May 13, 2022 •

edited

jakub-bochenski commented May 13, 2022 •

edited