curl / Mailing Lists / curl-library / Single Mail


Re: Point in time during HTTP 3xx response handling with CURLOPT_FOLLOWLOCATION when Location headers are followed

From: Nicolas Roeser via curl-library <>
Date: Mon, 15 Apr 2019 16:04:15 +0200


On 2019-04-07 at 23:51+02:00, Daniel Stenberg wrote:
> On Sun, 7 Apr 2019, Nicolas Roeser via curl-library wrote:
>> But for now I can not understand the necessity of reading the HTTP
>> response body for a 302 response _at all_(*) when there is a usable
>> Location header field.
> curl doesn't *have to*, but it choses to. Keeping the connection alive
> and reading a little extra data over it, is usually much faster than
> tearing down the connection and creating a new for next request.

I agree.

> If this is done over HTTP/2 (or the coming HTTP/3), the 302 request
> should be aborted since we can do that and still maintain the connection
> but I'm pretty sure this optimization is still left to implement.


>> Or am I using the wrong means for accomplishing my goal described
>> above? Do I need to implement redirection following myself?
> That could be a way to force this behavior you want, sure. But depending
> on many conditions, that's not necessarily the faster way.

Agreed. I have not implemented redirection following, but still use the
logic in libcurl for that.

I had also written:
>> I have not tried it yet, but could think of using a header callback function which monitors the headers for 3xx responses and for Location header fields. As soon as _both_ of them have be seen, I could set a flag which modifies the progress function logic until a new HTTP response is seen (which starts with "HTTP/"). This will still download the first big response payload entirely, won’t it?

I have decided to use this proposed way of implementing the code. The
progress function checks whether a redirecting resource (3xx header with
Location header field) is being handled, and if yes, raises the download
size limit.

(By the way, my progress function also peeks at a possible Content-Type
header field in order to abort the transfer if the document is not
interesting (in my case, not an HTML document) *and* is the last in a
(possible) chain of redirections.)

>> (*) Having thought about it a little more – is this behavior related
>> to keep-alive connections and not wanting to close them/open new ones?
> Yes it is exactly that!

Thanks for your explanations, they were very useful!

Nicolas Roeser
kiz – Information Systems Department, Ulm University
Received on 2019-04-15