cURL / Mailing Lists / curl-users / Single Mail

curl-users

Problem with chunked encoding/decoding

From: COMPAN Arnaud <compan_at_ipanematech.com>
Date: Thu, 19 Apr 2012 15:56:16 +0000

Hello,

I'm posting some data (about 2kB) using chunked encoding and the server responds with few data using chunked encoding also:

$ ./curl-7.25.0/src/curl --trace - -d @medium.txt --header "Transfer-Encoding: chunked" http://172.16.19.0:8080
== Info: About to connect() to 172.16.19.0 port 8080 (#0)
== Info: Trying 172.16.19.0...
== Info: connected
== Info: Connected to 172.16.19.0 (172.16.19.0) port 8080 (#0)
=> Send header, 243 bytes (0xf3)
0000: 50 4f 53 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d POST / HTTP/1.1.
0010: 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 63 75 72 .User-Agent: cur
0020: 6c 2f 37 2e 32 35 2e 30 20 28 69 36 38 36 2d 70 l/7.25.0 (i686-p
0030: 63 2d 6c 69 6e 75 78 2d 67 6e 75 29 20 6c 69 62 c-linux-gnu) lib
0040: 63 75 72 6c 2f 37 2e 32 35 2e 30 20 4f 70 65 6e curl/7.25.0 Open
0050: 53 53 4c 2f 30 2e 39 2e 38 6b 20 7a 6c 69 62 2f SSL/0.9.8k zlib/
0060: 31 2e 32 2e 33 2e 33 0d 0a 48 6f 73 74 3a 20 31 1.2.3.3..Host: 1
0070: 37 32 2e 31 36 2e 31 39 2e 30 3a 38 30 38 30 0d 72.16.19.0:8080.
0080: 0a 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 54 72 .Accept: */*..Tr
0090: 61 6e 73 66 65 72 2d 45 6e 63 6f 64 69 6e 67 3a ansfer-Encoding:
00a0: 20 63 68 75 6e 6b 65 64 0d 0a 43 6f 6e 74 65 6e chunked..Conten
00b0: 74 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 t-Type: applicat
00c0: 69 6f 6e 2f 78 2d 77 77 77 2d 66 6f 72 6d 2d 75 ion/x-www-form-u
00d0: 72 6c 65 6e 63 6f 64 65 64 0d 0a 45 78 70 65 63 rlencoded..Expec
00e0: 74 3a 20 31 30 30 2d 63 6f 6e 74 69 6e 75 65 0d t: 100-continue.
00f0: 0a 0d 0a ...
<= Recv header, 23 bytes (0x17)
0000: 48 54 54 50 2f 31 2e 31 20 31 30 30 20 43 6f 6e HTTP/1.1 100 Con
0010: 74 69 6e 75 65 0d 0a tinue..
== Info: Done waiting for 100-continue
=> Send data, 2142 bytes (0x85e)
0000: 38 35 37 0d 0a 58 30 30 30 30 30 30 30 30 30 30 857..X0000000000
0010: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0020: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0030: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0040: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0050: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0060: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0070: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0080: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0090: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
00a0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
00b0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
00c0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
00d0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
00e0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
00f0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0100: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0110: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0120: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0130: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0140: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0150: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0160: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0170: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0180: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0190: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
01a0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
01b0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
01c0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
01d0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
01e0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
01f0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0200: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0210: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0220: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0230: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0240: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0250: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0260: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0270: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0280: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0290: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
02a0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
02b0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
02c0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
02d0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
02e0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
02f0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0300: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0310: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0320: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0330: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0340: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0350: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0360: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0370: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0380: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0390: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
03a0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
03b0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
03c0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
03d0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
03e0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
03f0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0400: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0410: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0420: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0430: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0440: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0450: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0460: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0470: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0480: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0490: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
04a0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
04b0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
04c0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
04d0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
04e0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
04f0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0500: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0510: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0520: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0530: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0540: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0550: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0560: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0570: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0580: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0590: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
05a0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
05b0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
05c0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
05d0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
05e0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
05f0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0600: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0610: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0620: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0630: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0640: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0650: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0660: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0670: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0680: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0690: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
06a0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
06b0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
06c0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
06d0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
06e0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
06f0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0700: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0710: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0720: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0730: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0740: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0750: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0760: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0770: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0780: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0790: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
07a0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
07b0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
07c0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
07d0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
07e0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
07f0: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0800: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0810: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0820: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0830: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0840: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000
0850: 30 30 30 30 30 30 30 30 30 30 30 58 0d 0a 00000000000X..
=> Send data, 5 bytes (0x5)
0000: 30 0d 0a 0d 0a 0....
<= Recv header, 17 bytes (0x11)
0000: 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0010: 0a .
<= Recv header, 35 bytes (0x23)
0000: 53 65 72 76 65 72 3a 20 42 61 73 65 48 54 54 50 Server: BaseHTTP
0010: 2f 30 2e 36 20 50 79 74 68 6f 6e 2f 33 2e 32 2e /0.6 Python/3.2.
0020: 32 0d 0a 2..
<= Recv header, 37 bytes (0x25)
0000: 44 61 74 65 3a 20 54 68 75 2c 20 31 39 20 41 70 Date: Thu, 19 Ap
0010: 72 20 32 30 31 32 20 31 35 3a 33 36 3a 31 35 20 r 2012 15:36:15
0020: 47 4d 54 0d 0a GMT..
<= Recv header, 25 bytes (0x19)
0000: 43 61 63 68 65 2d 43 6f 6e 74 72 6f 6c 3a 20 6e Cache-Control: n
0010: 6f 2d 63 61 63 68 65 0d 0a o-cache..
<= Recv header, 40 bytes (0x28)
0000: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 70 Content-Type: ap
0010: 70 6c 69 63 61 74 69 6f 6e 2f 6f 63 74 65 74 2d plication/octet-
0020: 73 74 72 65 61 6d 0d 0a stream..
<= Recv header, 28 bytes (0x1c)
0000: 54 72 61 6e 73 66 65 72 2d 45 6e 63 6f 64 69 6e Transfer-Encodin
0010: 67 3a 20 63 68 75 6e 6b 65 64 0d 0a g: chunked..
<= Recv data, 14 bytes (0xe)
0000: 58 4f 4f 4f 4f 4f 58 0d 0a 30 0d 0a 0d 0a XOOOOOX..0....
== Info: Problem (2) in the Chunked-Encoded data
== Info: Closing connection #0
curl: (56) Problem (2) in the Chunked-Encoded data

As you can see, curl reports an error when decoding the answer.
Looking at the curl's traces, it's normal: the answer lacks a \r\n at the end of the headers (before the first chunk).
But the traces are wrong. Here is what wireshark has captured (full capture is attached):

0000 48 54 54 50 2f 31 2e 31 20 31 30 30 20 43 6f 6e HTTP/1.1 100 Con
0010 74 69 6e 75 65 0d 0a 48 54 54 50 2f 31 2e 31 20 tinue..H TTP/1.1
0020 32 30 30 20 4f 4b 0d 0a 53 65 72 76 65 72 3a 20 200 OK.. Server:
0030 42 61 73 65 48 54 54 50 2f 30 2e 36 20 50 79 74 BaseHTTP /0.6 Pyt
0040 68 6f 6e 2f 33 2e 32 2e 32 0d 0a 44 61 74 65 3a hon/3.2. 2..Date:
0050 20 54 68 75 2c 20 31 39 20 41 70 72 20 32 30 31 Thu, 19 Apr 201
0060 32 20 31 35 3a 33 36 3a 31 35 20 47 4d 54 0d 0a 2 15:36: 15 GMT..
0070 43 61 63 68 65 2d 43 6f 6e 74 72 6f 6c 3a 20 6e Cache-Co ntrol: n
0080 6f 2d 63 61 63 68 65 0d 0a 43 6f 6e 74 65 6e 74 o-cache. .Content
0090 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 69 -Type: a pplicati
00a0 6f 6e 2f 6f 63 74 65 74 2d 73 74 72 65 61 6d 0d on/octet -stream.
00b0 0a 54 72 61 6e 73 66 65 72 2d 45 6e 63 6f 64 69 .Transfe r-Encodi
00c0 6e 67 3a 20 63 68 75 6e 6b 65 64 0d 0a 0d 0a 37 ng: chun ked....7
00d0 0d 0a 58 4f 4f 4f 4f 4f 58 0d 0a 30 0d 0a 0d 0a ..XOOOOO X..0....

Note that if we post less data (a few bytes instead of a few kB), it works.

Thanks for your help,
                Arnaud

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html

Received on 2012-04-19