Hello,

I have run into an issue in libcurl where the uploaded size obtained
via CURLINFO_SIZE_UPLOAD is not measured correctly if the Expect
header is removed.

The cause appears to be as follows:

If it falls under the following conditions in Curl_http of http.c,

--------------
  // http.c ~L2751
  if(conn->httpversion != 20 &&
    !data->state.expect100header &&
    (postsize < MAX_INITIAL_POST_SIZE)) { // MAX_INITIAL_POST_SIZE = 64KB
--------------

Part of the data (about the size of the send buffer) is sent in this
function. The remaining data is sent in readwrite_upload() in
transfer.c.

At this time, only the size transmitted within readwrite_upload() is
counted as the upload size.

The following change in http.c seems to fix the issue, but of course
others that know the codebase much better may have a better
suggestion.

--------------
  // http.c ~L2879
  if(http->writebytecount >= postsize) {
    /* already sent the entire request body, mark the "upload" as
       complete */
    infof(data, "upload completely sent off: %" CURL_FORMAT_CURL_OFF_T
          " out of %" CURL_FORMAT_CURL_OFF_T " bytes\n",
          http->writebytecount, postsize);
    data->req.upload_done = TRUE;
    data->req.keepon &= ~KEEP_SEND; /* we're done writing */
    data->req.exp100 = EXP100_SEND_DATA; /* already sent */
    Curl_expire_done(data, EXPIRE_100_TIMEOUT);
  }
+ else {
+ data->req.writebytecount = http->writebytecount;
+ }
--------------

Now for a simple reproduction example:

--------------

#include <curl/curl.h>

char g_Data[40 * 1024]; // POST 40KB

int sockopt_callback(void *clientp, curl_socket_t curlfd, curlsocktype purpose)
{
    int sndbufsize = 4 * 1024; // 4KB send buffer
    setsockopt(curlfd, SOL_SOCKET, SO_SNDBUF,
           (const char *)&sndbufsize, sizeof(sndbufsize));

    return CURL_SOCKOPT_OK;
}

int main(int argc, char *argv[])
{
    curl_slist* pHeaderList = NULL;
    CURL* pCurl = curl_easy_init();

    curl_easy_setopt(pCurl, CURLOPT_SOCKOPTFUNCTION, sockopt_callback);

    curl_easy_setopt(pCurl, CURLOPT_CUSTOMREQUEST, "POST");
    curl_easy_setopt(pCurl, CURLOPT_URL, "http://example.com");

    curl_easy_setopt(pCurl, CURLOPT_POSTFIELDS, g_Data);
    curl_easy_setopt(pCurl, CURLOPT_POSTFIELDSIZE, sizeof (g_Data));

    // Remove "Expect: 100-continue"
    pHeaderList = curl_slist_append(pHeaderList, "Expect:");

    curl_easy_setopt(pCurl, CURLOPT_HTTPHEADER, pHeaderList);

    CURLcode code = curl_easy_perform(pCurl);

    if (code == CURLE_OK)
    {
        double uploadSize = 0.0;
        curl_easy_getinfo(pCurl, CURLINFO_SIZE_UPLOAD, &uploadSize);

        printf("uploadSize = %f\n", uploadSize);

        if (static_cast<size_t>(uploadSize) == sizeof (g_Data))
        {
            printf("!!!!!!!!!! PASS\n");
        }
        else
        {
            printf("!!!!!!!!!! FAIL\n");
        }
    }
    else
    {
        printf("curl_easy_perform() failed. e = %d\n", code);
    }

    curl_slist_free_all(pHeaderList);
    curl_easy_cleanup(pCurl);

    return 0;
}

--------------

Expected result:

uploadSize = 40960.000000
!!!!!!!!!! PASS

Actual result:

uploadSize = 36703.000000
!!!!!!!!!! FAIL

The upload size ends up being roughly the size of the POST minus the
size of the send buffer. And please note that the entire buffer does
in fact get POSTed. It is only reported incorrectly by
CURLINFO_SIZE_UPLOAD.

Keys to reproducing this issue:
- Must POST less than MAX_INITIAL_POST_SIZE (64KB).
- Must set send buffer socket option to much less than amount POSTing
(e.g. 4KB).
- Must remove Expect header.
- Socket must be non-blocking (default).

System information:
- FreeBSD 11.1 (I would expect this to occur on most OSes, not
including Windows)
- libcurl 7.61.0
- URL: http://example.com

Thank you for your time and I will do my best to answer any questions
you may have.
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2018-08-08