Bugs item #1531838, was opened at 2006-07-31 18:11
Message generated for change (Comment added) made by bagder
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1531838&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
>Group: bad behaviour
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: Mark Lentczner (mark_lentczner)
>Assigned to: Daniel Stenberg (bagder)
Summary: uploads with chunked encoding broken if header specified

Initial Comment:
Consider this sequence:

curl_easy_setopt(new_req->mCurlHandle, CURLOPT_UPLOAD, 1);

headers = curl_slist_append(headers, "Transfer-Encoding: chunked");
curl_easy_setopt(mCurlHandle, CURLOPT_HTTPHEADER, headers); if

curl_easy_setopt(new_req->mCurlHandle, CURLOPT_READFUNCTION,
                    &myUploadCallback);

// don't set CURLOPT_INFILESIZE

This causes libCURL to announce the request body (the upload) as
chunked, but then to not chunk encode the buffers from the callback.

The bug is in http.c, starting with the line:

if(!conn->bits.upload_chunky && (httpreq != HTTPREQ_GET)) {
...
}
else if(conn->bits.upload_chunky) {
    ....
    if(!checkheaders(data, "Transfer-Encoding:")) {
        te = "Transfer-Encoding: chunked\r\n";
    }
    else {
        te = "";
        conn->bits.upload_chunky = FALSE;
    }
}

As currently written, the logic runs like this:

if the upload is chunk (as the CURLOPT_INFILESIZE is not specified)
  if there is no transfer-encoding header
    set the transfer-encoding to chunked
  else (there is a transfer-encoding header
    set the transfer-encoding to ""
    turn off chunked

This second if should probably be something like:

ptr = checkheaders(data, "Transfer-Encoding:");
if(!ptr || Curl_compareheader(ptr, "Transfer-Encoding:", "chunked") {

So the logic is more like:

  if there no transfer-encoding header, or if it is chunked
    set the transfer-encoding to chunked
  else (there is a transfer-encoding header and it isn't chunked)
    set the transfer-encoding to ""
    turn off chunked

markl_at_lindenlab.com

----------------------------------------------------------------------

>Comment By: Daniel Stenberg (bagder)
Date: 2006-08-02 20:20

Message:
Logged In: YES
user_id=1110

Many thanks for your report and your patch.

I've applied and committed.

----------------------------------------------------------------------

Comment By: Mark Lentczner (mark_lentczner)
Date: 2006-08-02 04:26

Message:
Logged In: YES
user_id=219202

a patch that fixes this issue is now attached. This patch might look like it
does too much, but it doesn't. Once one fixes the heart of the bug (that the
old code assumes, in one branch of the if, that any user supplied transfer-
encoding header invalidates chunked, without checked if it is indeed
chunked), then you can see that the whole structure collapses down to a
much simpler set of tests.

Furthermore, the test for GET is irrelevant, as upload_chunked is only set if in
an upload, and in that case the httpreq variable is forced to PUT (see later in
http.c where this is done).

----------------------------------------------------------------------

Comment By: Mark Lentczner (mark_lentczner)
Date: 2006-08-01 15:03

Message:
Logged In: YES
user_id=219202

You can easily reproduce the bug simply using command line curl:

U=http://www.ozonehouse.com/mark/test-sl/foo2.pl
echo bar | curl -T - -v $U
echo bar | curl -T - -H 'Transfer-Encoding: chunked' -v $U

(Note: the URL doesn't really matter, it just needs to be someplace that will
accept chunked request bodies --the URL shown is on my server, you are free
to use it.)

Looking at the -v output, you'll see that both are identical, and both use
chunked transfer encoding for the reuqest body. (Note: the second command
will never complete, you'll have to control-C it).

Now, if you run tcpdump while running the commands you can see the
problem. With the first command, the first packet from curl after the 100
Continue response is a properly formatted chunk:

        0x0030: 0c71 ccd8 340d 0a62 6172 0a0d 0a .q..4..bar...

That's 4 bytes ("bar\n") and then the data and a trailing \r\n.

With the second command, the first packet from curl after the 100 Continue
response is not formatted as a chunk, it is just the raw data:

        0x0030: 0c72 e8c7 6261 720a .r..bar.

When Apache is tries to interpret this as a chunk response it seems to hang
waiting for more data (understandable...)

Notice that the only difference between the two curl commands is that the
second one explicitly adds the Transfer-Encoding: chunked header that curl
was going to add anyway (because the input length is unknown).

Finally, I uncovered this coding to libcurl, so it isn't a bug in the command
line curl utility -- it was just easier to demonstrate that way.

I will try to work on a patch fix later... but you may be able to figure it out
from the above...

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2006-07-31 23:18

Message:
Logged In: YES
user_id=1110

Thanks for your report!

Can you please:

1. tell us a way to repeat this problem?

2. show us your suggested fix with a diff -u output? Your
description in here is a tad bit hard to follow.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1531838&group_id=976
Received on 2006-08-02