Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CURLOPT_READFUNCTION not called following empty POST #10313

Closed
RanBarLavie opened this issue Jan 17, 2023 · 4 comments
Closed

CURLOPT_READFUNCTION not called following empty POST #10313

RanBarLavie opened this issue Jan 17, 2023 · 4 comments

Comments

@RanBarLavie
Copy link

Problem Description

My application uses libcurl to send two POST requests, one after the other, over the same connection.

  1. The 1st one is a POST request with no content (Content-Length: 0).
  2. The 2nd is a "chunked" Transfer-Encoding POST request, which supplies the content with the CURLOPT_READFUNCTION callback.

The 1st request goes through and receives a response, as expected.
For some reason, the 2nd request only sends out the header. It seems libcurl never calls the callback function to consume the content.

Reproduce

The problem can be reproduced like this:

#include <curl/curl.h>
#include <assert.h>

static size_t read_callback(char* buffer, size_t size, size_t nitems, void* userdata);

int main(int argc, char* argv[])
{
  CURL* curl = curl_easy_init();
  assert(curl != NULL);
  const char* url = argc > 1 ? argv[1] : "http://localhost:8080/";
  curl_easy_setopt(curl, CURLOPT_URL, url);
  curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

  /* 1st request is a POST with no content */
  curl_easy_setopt(curl, CURLOPT_POST, 1L);
  curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, 0L);
  curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "");
  CURLcode result = curl_easy_perform(curl);
  assert(result == CURLE_OK);

  /* 2nd request is a POST with "chunked" transfer encoding */
  curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);
  curl_easy_setopt(curl, CURLOPT_POSTFIELDS, NULL);
  struct curl_slist* slist = curl_slist_append(NULL, "Transfer-Encoding: chunked");
  curl_easy_setopt(curl, CURLOPT_HTTPHEADER, slist);
  result = curl_easy_perform(curl);
  assert(result == CURLE_OK);

  curl_slist_free_all(slist);
  curl_easy_cleanup(curl);

  return 0;
}

static size_t read_callback(char* buffer, size_t size, size_t nitems, void* userdata)
{
  static int done = 0;
  if (done) {
    return 0;
  }
  done = 1;
  assert(size * nitems >= 2);
  buffer[0] = 'H';
  buffer[1] = 'i';
  return 2;
}

Output:

./curltest http://httpbin.org/post
* About to connect() to httpbin.org port 80 (#0)
*   Trying 34.205.150.168...
* Connected to httpbin.org (34.205.150.168) port 80 (#0)
> POST /post HTTP/1.1
Host: httpbin.org
Accept: */*
Content-Length: 0
Content-Type: application/x-www-form-urlencoded

< HTTP/1.1 200 OK
< (...response truncated for conciseness...)
* Connection #0 to host httpbin.org left intact
* Found bundle for host httpbin.org: 0x1e56ab0
* Re-using existing connection! (#0) with host httpbin.org
* Connected to httpbin.org (34.205.150.168) port 80 (#0)
> POST /post HTTP/1.1
Host: httpbin.org
Accept: */*
Transfer-Encoding: chunked
Content-Type: application/x-www-form-urlencoded

And that's where program execution hangs, instead of sending out the message body.

Debug:
reproduce.txt

Workaround

Interestingly, the problem is eliminated by changing the 1st request to send some content. Even 1 byte does the trick, like this for example:

  curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, 1L);
  curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "X");

The fact that the problem (in the 2nd request) is fixed just by setting some content in the body of the 1st request makes this look like a bug.

This is the output with the workaround in place:
workaround.txt

Version

libcurl version: 7.87.0

[curl -V output]

# curl -V
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.87.0 OpenSSL/1.0.2k-fips zlib/1.2.7 nghttp2/1.31.1
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS Debug TrackMemory IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

Operating System

# uname -a
Linux blade-162-115 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Any insight will be appreciated. I will be happy to provide more information as needed.
Thank you!

@dfandrich
Copy link
Contributor

dfandrich commented Jan 17, 2023 via email

@RanBarLavie
Copy link
Author

Thanks @dfandrich for looking into this and replying so fast.
The reason CURLOPT_POSTFIELDSIZE remains 0 in the 2nd request is that it's a "chunked" transfer encoding request, which does not know the content length in advance.
I realize that setting CURLOPT_POSTFIELDSIZE=2 in the 2nd request would eliminate the problem in the test app. But it's not clear what a "real-world" application can do in this case, if it doesn't know the payload size in advance.

@dfandrich
Copy link
Contributor

dfandrich commented Jan 17, 2023 via email

@RanBarLavie
Copy link
Author

Yes! That solves it.
Now I see that -1 is the default value for CURLOPT_POSTFIELDSIZE, so it all makes sense.

The documentation is a bit confusing here - CURLOPT_POSTFIELDSIZE only states that -1 will cause libcurl to strlen the data, and not mentioning its impact on the read callback.

So, I'll do a bit more testing and then close this issue.
Thank you @dfandrich !

dfandrich added a commit that referenced this issue Jan 17, 2023
Reported-by: RanBarLavie on github

Closes #10313
dfandrich added a commit that referenced this issue Jan 20, 2023
Reported-by: RanBarLavie on github

Closes #10313
bch pushed a commit to bch/curl that referenced this issue Jul 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging a pull request may close this issue.

3 participants