Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Curl doesn't POST full text file if it contains old Mac-style CR line endings #13063

Closed
simon-20 opened this issue Mar 6, 2024 · 3 comments
Closed

Curl doesn't POST full text file if it contains old Mac-style CR line endings #13063

simon-20 opened this issue Mar 6, 2024 · 3 comments
Assignees
@bagder
Labels
cmdline tool

Comments

@simon-20
Copy link

simon-20 commented Mar 6, 2024

I did this

When using curl to POST a text file containing CR line endings, or containing mixed line endings (CR and CRLF), curl does not upload the whole file, but truncates it.

Steps to reproduce:

Download the attached file, which contains CR line endings. (Verify with a hex editor, if needed; note that opening this file in many IDEs will automatically convert it to CRLF or LF format).

POST the file to an HTTP echo debugging service, e.g.:

curl -X POST https://echo.free.beeceptor.com --header "Content-Type: application/xml" --data @xml-with-cr-endings.txt

Observe that when using --data the rawBody returned by the echo service is not the complete file. This is not down to the echo service; you can verify using a different one (e.g. Postman), or using a local API mocker.

By contrast, when using:

curl -X POST https://echo.free.beeceptor.com --header "Content-Type: application/xml" --data-binary @xml-with-cr-endings.txt

The rawBody returned by the echo service is the complete file.

xml-with-cr-endings.txt

I expected the following

Since CR is a valid line ending for text files, I would expect the whole file to be uploaded when using the --data flag; I should not have to use the --data-binary flag.

(CR is the line ending that used to be used on Mac OS, before the switch to a Unix-based platform. It is still used in many industrial / health / elecontrics applications. Such line endings are valid in the XML standard, although the standard stipulates that the XML parser should normalise line endings: https://www.w3.org/TR/REC-xml/#sec-line-ends).

curl/libcurl version

curl 8.5.0 (x86_64-conda-linux-gnu) libcurl/8.5.0 OpenSSL/3.2.1 zlib/1.2.13 zstd/1.5.5 libssh2/1.11.0 nghttp2/1.58.0

I also experienced the problem using curl 7.81, which is the version of curl currently available in the apt package manager for Ubuntu 22.

operating system

Linux 6.5.0-21-generic #21~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 9 13:32:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
@simon-20
Copy link
Author

simon-20 commented Mar 6, 2024

(I posted this here because it seems like a bug, but not really a security problem in curl itself, although it could potentially be used to exploit systems which use curl to upload text-based user-submitted files, if those systems verify the integrity of the text files taking into consideration CR line endings, and then use curl to upload the data. Anyway, hope it's okay here).

@bagder bagder self-assigned this Mar 6, 2024
@bagder
Copy link
Member

bagder commented Mar 6, 2024

Kind of fascinating that it has had this flaw since at least 1998 and you are the first to notice.

bagder added a commit that referenced this issue Mar 6, 2024
@bagder
paramhlp: fix CRLF-stripping files with "-d @file"
0de0905 
All CR and LF bytes should be stripped, as documented, and all other
bytes are inluded in the data. Starting now, it also excludes null bytes
as they would otherwise also cut the data short.

Reported-by: Simon K
Fixes #13063
@bagder bagder mentioned this issue Mar 6, 2024
Closed
@simon-20
Copy link
Author

simon-20 commented Mar 6, 2024

Yes, that is fascinating. Thank you for fixing it so quickly.

bagder added a commit that referenced this issue Mar 6, 2024
@bagder
paramhlp: fix CRLF-stripping files with "-d @file"
09fc2cd 
All CR and LF bytes should be stripped, as documented, and all other
bytes are inluded in the data. Starting now, it also excludes null bytes
as they would otherwise also cut the data short.

Reported-by: Simon K
Fixes #13063
Closes #13064
@bagder bagder closed this as completed in 923f7f8 Mar 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bagder bagder

Labels
cmdline tool
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

paramhlp: fix CRLF-stripping files with "-d @file" curl/curl
2 participants
@bagder @simon-20