Hello,

TLDR; curl seem to be adding extra data to the payload when using
`--upload-file -` together with `--header "Content-Length: XXX"`.

The below is a reduced example. In reality we need to upload a file,
which is very large, whose size we know, but that is not available
locally on disk. Therefore we pass the contents to upload to curl via
STDIN and we set the (known) content length header.

So here is the reduced example that demonstrates the bug/problem:

        $ curl -V
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10
zlib/1.2.8 libidn/1.32 librtmp/2.3
        Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps
pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
        Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM
NTLM_WB SSL libz TLS-SRP UnixSockets

        $ lsb_release -d
        Description: Ubuntu 16.04.1 LTS

        # File to send
        $ cat /tmp/hello_world.txt
        Hello, my name is Foooooo

        # Size of the file
        $ cat /tmp/hello_world.txt | wc -c
        26

        # Do send file via stdin via curl
        $ cat /tmp/hello_world.txt | \
          curl --trace /tmp/trace \
               --upload-file - \
               --request POST \
               --user secret:secret \
               --header 'Content-Type: application/octet-stream' \
               --header 'Content-Length: 26' \
               https://example.org/secret

Now, if I look at the /tmp/trace (redacted) I see:

$ cat /tmp/trace
[..SSL handshake etc...]
=> Send header, 333 bytes (0x14d)
0000: 50 4f 53 54 20 2f XX XX XX XX XX XX XX XX XX XX POST /XXXXXXXXXX
0010: XX XX XX XX XX XX XX XX XX XX XX 2f 20 48 54 54 XXXXXXXXXXX/ HTT
0020: 50 2f 31 2e 31 0d 0a 48 6f 73 74 3a 20 64 69 72 P/1.1..Host: XXX
0030: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
0040: XX XX XX XX XX XX 0d 0a 41 75 74 68 6f 72 69 7a XXXXXX..Authoriz
0050: 61 74 69 6f 6e 3a 20 42 61 73 69 63 20 XX XX XX ation: Basic XXX
0060: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
0070: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
0080: XX XX XX XX XX XX XX XX XX XX XX XX XX 0d 0a 55 XXXXXXXXXXXXX..U
0090: 73 65 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c 2f ser-Agent: curl/
00a0: 37 2e 34 37 2e 30 0d 0a 41 63 63 65 70 74 3a 20 7.47.0..Accept:
00b0: 2a 2f 2a 0d 0a 54 72 61 6e 73 66 65 72 2d 45 6e */*..Transfer-En
00c0: 63 6f 64 69 6e 67 3a 20 63 68 75 6e 6b 65 64 0d coding: chunked.
00d0: 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 .Content-Type: a
00e0: 70 70 6c 69 63 61 74 69 6f 6e 2f 6f 63 74 65 74 pplication/octet
00f0: 2d 73 74 72 65 61 6d 0d 0a 43 6f 6e 74 65 6e 74 -stream..Content
0100: 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 69 -Type: applicati
0110: 6f 6e 2f 6f 63 74 65 74 2d 73 74 72 65 61 6d 0d on/octet-stream.
0120: 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a .Content-Length:
0130: 20 32 36 0d 0a 45 78 70 65 63 74 3a 20 31 30 30 26..Expect: 100
0140: 2d 63 6f 6e 74 69 6e 75 65 0d 0a 0d 0a -continue....
<= Recv header, 23 bytes (0x17)
0000: 48 54 54 50 2f 31 2e 31 20 31 30 30 20 43 6f 6e HTTP/1.1 100 Con
0010: 74 69 6e 75 65 0d 0a tinue..
=> Send data, 32 bytes (0x20)
0000: 31 61 0d 0a 48 65 6c 6c 6f 2c 20 6d 79 20 6e 61 1a..Hello, my na
0010: 6d 65 20 69 73 20 46 6f 6f 6f 6f 6f 6f 0a 0d 0a me is Foooooo...
=> Send data, 5 bytes (0x5)
0000: 30 0d 0a 0d 0a 0....
<= Recv header, 17 bytes (0x11)
0000: 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0010: 0a .
<= Recv header, 28 bytes (0x1c)
0000: 54 72 61 6e 73 66 65 72 2d 45 6e 63 6f 64 69 6e Transfer-Encodin
0010: 67 3a 20 63 68 75 6e 6b 65 64 0d 0a g: chunked..
<= Recv header, 37 bytes (0x25)
0000: 44 61 74 65 3a 20 4d 6f 6e 2c 20 30 32 20 4a 61 Date: Mon, 02 Ja
0010: 6e 20 32 30 31 37 20 31 30 3a 34 35 3a 35 32 20 n 2017 10:45:52
0020: 47 4d 54 0d 0a GMT..
<= Recv header, 25 bytes (0x19)
0000: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 Content-Type: te
0010: 78 74 2f 68 74 6d 6c 0d 0a xt/html..
<= Recv header, 27 bytes (0x1b)
0000: 53 65 72 76 65 72 3a 20 54 77 69 73 74 65 64 57 Server: TwistedW
0010: 65 62 2f 31 32 2e 31 2e 30 0d 0a eb/12.1.0..
<= Recv header, 2 bytes (0x2)
0000: 0d 0a ..
<= Recv data, 48 bytes (0x30)
0000: 32 35 0d 0a XX XX XX XX XX XX XX XX XX XX XX XX 25..XXXXXXXXXXXX
0010: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
0020: XX XX XX XX XX XX XX XX 0a 0d 0a 30 0d 0a 0d 0a XXXXXXXXX...0....
== Info: Connection #0 to host XXXXXXXXXXXXXXXXXXXXXXXXX left intact

You'll note in the trace above there's "31 61 0d 0a" preceeding the
"Hello, my name is...".

I also can confirm that on the receiving side, the file that is saved on
the server also has the values "31 61 0d 0a" added in front of the
original text.

Now my questions are:
* is this me using curl the wrong way?
* what would be the correct way to upload a file via stdin?
* is this a curl bug?

Thanks,
*t
-------------------------------------------------------------------
List admin: https://cool.haxx.se/list/listinfo/curl-users
FAQ: https://curl.haxx.se/docs/faq.html
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2017-01-02