Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

http fails on 416 when resuming #10521

Closed
Smackd0wn opened this issue Feb 15, 2023 · 2 comments
Closed

http fails on 416 when resuming #10521

Smackd0wn opened this issue Feb 15, 2023 · 2 comments
Labels
HTTP

Comments

@Smackd0wn
Copy link

Smackd0wn commented Feb 15, 2023 •
edited

I did this

On git master, curl --fail -O -C - http://host/file when the local file had already been completely downloaded.

Server response:

< HTTP/1.1 416 Requested Range Not Satisfiable
< Server: nginx/1.22.1
< Date: Wed, 15 Feb 2023 13:29:29 GMT
< Content-Type: text/html; charset=utf8
< Content-Length: 197
< Content-Range: bytes */1234567

I expected the following

curl exits with code 0 (no error, even with --fail), because in http_should_fail():

curl/lib/http.c

Lines 1161 to 1167 in 3027611

/*
** A 416 response to a resume request is presumably because the file is
** already completely downloaded and thus not actually a fail.
*/
if(data->state.resume_from && data->state.httpreq == HTTPREQ_GET &&
httpcode == 416)
return FALSE;

However, when the requested range was unsatisfied (e.g., Content-Range: */1234567), data->state.resume_from is overwritten with 0 (see line 3479), and thus the check above always fail.

curl/lib/http.c

Lines 3452 to 3480 in 3027611

else if(!k->http_bodyless && checkprefix("Content-Range:", headp)) {
/* Content-Range: bytes [num]-
Content-Range: bytes: [num]-
Content-Range: [num]-
Content-Range: [asterisk]/[total]
The second format was added since Sun's webserver
JavaWebServer/1.1.1 obviously sends the header this way!
The third added since some servers use that!
The fourth means the requested range was unsatisfied.
*/
char *ptr = headp + strlen("Content-Range:");
/* Move forward until first digit or asterisk */
while(*ptr && !ISDIGIT(*ptr) && *ptr != '*')
ptr++;
/* if it truly stopped on a digit */
if(ISDIGIT(*ptr)) {
if(!curlx_strtoofft(ptr, NULL, 10, &k->offset)) {
if(data->state.resume_from == k->offset)
/* we asked for a resume and we got it */
k->content_range = TRUE;
}
}
else
data->state.resume_from = 0; /* get everything */
}

@vszakats vszakats added the HTTP label Feb 16, 2023
@piru
Copy link

piru commented Feb 21, 2023

It appears nginx returns: content-range: bytes */<filesize> in this case. For example Apache2 doesn't include the content-range header for 416 response.

This seems to fix this issue, but I am not entirely sure if this is the best way to fix it:

diff --git a/lib/http.c b/lib/http.c
index ee31aa2c6..caa3d50a1 100644
--- a/lib/http.c
+++ b/lib/http.c
@@ -3476,7 +3476,7 @@ CURLcode Curl_http_header(struct Curl_easy *data, struct connectdata *conn,
           k->content_range = TRUE;
       }
     }
-    else
+    else if(416 != k->httpcode)
       data->state.resume_from = 0; /* get everything */
   }
 #if !defined(CURL_DISABLE_COOKIES)

Comments are welcome.

piru pushed a commit to piru/curl that referenced this issue Feb 28, 2023
http: ignore content-range for 416 responses
c3867e9 
416 Requested Range Not Satisfiable returned by some HTTP daemons
(nginx at least) appear to include Content-Range header. This
confuses curl and makes it think that the whole file needs to be
downloaded again, resulting in download failure when in fact the
while file has already been downloaded.

This change makes curl disregard Content-Range header on a 416
response, allowing other existing logic consider the file fully
downloaded.

Fixes curl#10521
@piru piru mentioned this issue Feb 28, 2023
Closed
@akabanovs akabanovs mentioned this issue Mar 8, 2023
Merged
@hmeine
Copy link

hmeine commented Aug 9, 2023

I run into this problem as well. I tried running curl -C - to download a build artifact from an artifactory server, hoping that repeated script runs would not download the large file multiple times.

The server replies with

HTTP/1.1 416 Requested Range Not Satisfiable
Connection: keep-alive
Content-Length: 0
Content-Range: bytes */977880575
Date: Wed, 09 Aug 2023 09:57:07 GMT
Server: nginx/1.18.0 (Ubuntu)
X-Artifactory-Id: ***redacted***
X-Artifactory-Node-Id: ***redacted***
X-Jfrog-Version: Artifactory/7.55.13 75513900

and curl fails (exit code 22).

As I am not familiar with the curl codebase, I have problems reviewing the change in #10644 and do not yet understand what is the problem with it.

@hmeine hmeine mentioned this issue Aug 9, 2023
Open
@Smackd0wn Smackd0wn mentioned this issue Oct 22, 2023
Closed
@bagder bagder mentioned this issue Oct 26, 2023
Closed
@bagder bagder closed this as completed in 225db91 Oct 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
HTTP
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

http: consider resume with CURLOPT_FAILONERRROR and 416 to be fine curl/curl
4 participants
@hmeine @piru @vszakats @Smackd0wn