curl-library
Re: Ranged PUTs, Content-Range, and Content-Length
Date: Sun, 5 May 2019 03:48:39 -0400
On 5/4/2019 4:04 AM, Ray Satiro wrote:
>
> Also as you noted CURLOPT_RANGE does not work like
> CURLOPT_RESUME_FROM, in that the former does not have any effect on
> Content-Length. Content-Length is determined by CURLOPT_INFILESIZE
> minus CURLOPT_RESUME_FROM. Yet if you specify CURLOPT_RESUME_FROM then
> CURLOPT_RANGE is ignored, and only CURLOPT_RESUME_FROM changes the
> starting offset of the file.
>
> If you use CURLOPT_RANGE and CURLOPT_INFILESIZE without
> CURLOPT_RESUME_FROM then libcurl ends up in this block [1]:
>
> else {
> /* Range was selected and then we just pass the incoming range and
> append total size */
> conn->allocptr.rangeline =
> aprintf("Content-Range: bytes %s/%" CURL_FORMAT_CURL_OFF_T
> "\r\n",
> data->state.range, data->state.infilesize);
> }
>
> But I think that will not work correctly since Content-Length is also
> CURLOPT_INFILESIZE. IIRC it is not good to set the Content-Length
> header on your own because then internally libcurl doesn't change the
> length to 0 when it does authentication before the PUT. Therefore I
> think the best thing to do here is do not use CURLOPT_RANGE (at least
> until its use is clarified) and set the range header yourself, as I
> suggested. I will try experimenting with it some later.
>
I played around with it and the idea I proposed initially to add in your
offset to READFUNCTION and SEEKFUNCTION is not needed since you can use
libcurl to resume from an arbitrary position. You should still implement
a regular seek function and (if on windows) a regular read function.
Instead of setting INFILESIZE to the file's actual size you would set it
to the resume + part size (or the file's actual size if it goes over).
Here it is boiled down:
curl_off_t end = (filesize < (resume + partsize) ? filesize : (resume
+ partsize));
curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, end);
curl_easy_setopt(curl, CURLOPT_RESUME_FROM_LARGE, resume);
char range_header[256];
sprintf(range_header, "Content-Range: bytes %" CURL_FORMAT_CURL_OFF_T "-"
"%" CURL_FORMAT_CURL_OFF_T "/*",
resume, end - 1);
struct curl_slist *header_list = NULL;
header_list = curl_slist_append(NULL, range_header);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, header_list);
res = curl_easy_perform(curl);
if(end == filesize)
done;
For example say your input file size is 6 and your part size is 4 then
there would be two transfers.
Content-Range: bytes 0-3/*
Content-Length: 4
Content-Range: bytes 4-5/*
Content-Length: 2
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2019-05-05