Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTTP POST with CURLOPT_READFUNCTION takes a long time to complete on Windows, libcurl 7.71.0 #5631

Closed
tmkk opened this issue Jun 30, 2020 · 2 comments
Closed

HTTP POST with CURLOPT_READFUNCTION takes a long time to complete on Windows, libcurl 7.71.0 #5631

tmkk opened this issue Jun 30, 2020 · 2 comments
Labels
performance regression Windows Windows-specific

Comments

@tmkk
Copy link

tmkk commented Jun 30, 2020

I did this

Compiled and ran the following code:

#include <curl/curl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>

#define POSTSIZE 1*1024*1024
static char *postdata;
static int bytesLeft = POSTSIZE;
static double start_ms;

static size_t read_callback(char *buffer, size_t size, size_t nitems, void *userdata)
{
    struct timeval tv;
    gettimeofday(&tv, NULL);
    double now = tv.tv_sec*1e+3 + tv.tv_usec*1e-3;
    int i,bytesToCopy = bytesLeft;
    if(bytesToCopy > size*nitems) bytesToCopy = size*nitems;
    for(i=0;i<bytesToCopy;i++) *buffer++ = *(postdata+POSTSIZE-bytesLeft--);
    fprintf(stderr,"Filled %d/%d bytes, %.3f msec elapsed\n",POSTSIZE-bytesLeft,POSTSIZE,now-start_ms);
    return bytesToCopy;
}

int main(void)
{
    postdata = (char *)calloc(POSTSIZE, 1);
    CURL *curl = curl_easy_init();
    curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
    curl_easy_setopt(curl, CURLOPT_URL, "http://192.168.1.160/");
    curl_easy_setopt(curl, CURLOPT_POST, 1L);
    curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, POSTSIZE);
    curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);

    struct timeval tv;
    gettimeofday(&tv, NULL);
    start_ms = tv.tv_sec*1e+3 + tv.tv_usec*1e-3;

    curl_easy_perform(curl);

    return 0;
}

An upload buffer is filled by a callback set with CURLOPT_READFUNCTION, and the callback is called 16 times to fill 1 MiB of data to be uploaded. On libcurl version 7.71.0 it takes about a second between every callback function calls, and results in more than 10 seconds to complete.

% ./a.exe
*   Trying 192.168.1.160:80...
* Connected to 192.168.1.160 (192.168.1.160) port 80 (#0)
> POST / HTTP/1.1
Host: 192.168.1.160
Accept: */*
Content-Length: 1048576
Content-Type: application/x-www-form-urlencoded

Filled 65536/1048576 bytes, 8.994 msec elapsed
Filled 131072/1048576 bytes, 206.329 msec elapsed
Filled 196608/1048576 bytes, 1207.389 msec elapsed
Filled 262144/1048576 bytes, 2208.778 msec elapsed
Filled 327680/1048576 bytes, 3208.970 msec elapsed
Filled 393216/1048576 bytes, 4210.744 msec elapsed
Filled 458752/1048576 bytes, 5212.402 msec elapsed
Filled 524288/1048576 bytes, 6213.489 msec elapsed
Filled 589824/1048576 bytes, 7216.166 msec elapsed
Filled 655360/1048576 bytes, 8218.369 msec elapsed
Filled 720896/1048576 bytes, 9220.328 msec elapsed
Filled 786432/1048576 bytes, 10221.817 msec elapsed
Filled 851968/1048576 bytes, 11223.812 msec elapsed
Filled 917504/1048576 bytes, 12225.841 msec elapsed
Filled 983040/1048576 bytes, 13227.434 msec elapsed
Filled 1048576/1048576 bytes, 14228.673 msec elapsed
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK

I expected the following

On version 7.70.0 and earlier it finishes almost instantly.

% ./a.exe
*   Trying 192.168.1.160:80...
* Connected to 192.168.1.160 (192.168.1.160) port 80 (#0)
> POST / HTTP/1.1
Host: 192.168.1.160
Accept: */*
Content-Length: 1048576
Content-Type: application/x-www-form-urlencoded

Filled 65536/1048576 bytes, 21.023 msec elapsed
Filled 131072/1048576 bytes, 22.016 msec elapsed
Filled 196608/1048576 bytes, 24.014 msec elapsed
Filled 262144/1048576 bytes, 25.014 msec elapsed
Filled 327680/1048576 bytes, 26.013 msec elapsed
Filled 393216/1048576 bytes, 26.013 msec elapsed
Filled 458752/1048576 bytes, 26.013 msec elapsed
Filled 524288/1048576 bytes, 27.013 msec elapsed
Filled 589824/1048576 bytes, 27.013 msec elapsed
Filled 655360/1048576 bytes, 28.013 msec elapsed
Filled 720896/1048576 bytes, 28.013 msec elapsed
Filled 786432/1048576 bytes, 29.012 msec elapsed
Filled 851968/1048576 bytes, 31.010 msec elapsed
Filled 917504/1048576 bytes, 31.010 msec elapsed
Filled 983040/1048576 bytes, 32.010 msec elapsed
Filled 1048576/1048576 bytes, 33.009 msec elapsed
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK

curl/libcurl version

libcurl version 7.71.0, a commit 8bc25c5 and later

operating system

Tested on Windows 10 1909 build 18363.904, the test code is built with i686-w64-mingw32-gcc.
Windows only issue, not affected on other platforms.
@bagder
Copy link
Member

bagder commented Jun 30, 2020

Just to clarify, if you build with 7.71.0 with commit 8bc25c5 reverted, everything is fine again?

bagder added a commit that referenced this issue Jun 30, 2020
@bagder
Revert "multi: implement wait using winsock events"
9d744a9 
This reverts commit 8bc25c5.

That commit (from #5397) introduced a regression in 7.71.0.

Reported-by: tmkk on github
Fixes #5631
This was referenced Jun 30, 2020
Closed
Closed
@tmkk
Copy link
Author

tmkk commented Jun 30, 2020

Confirmed that it's fine when the commit 8bc25c5 is reverted from the 7.71.0 release.

@bagder bagder closed this as completed in 25222b1 Jun 30, 2020
@rcombs rcombs mentioned this issue Jun 30, 2020
Closed
@bagder bagder mentioned this issue Jun 30, 2020
Closed
@bagder bagder mentioned this issue Aug 5, 2020
Closed
@jay jay mentioned this issue Oct 30, 2020
Closed
@tmkk tmkk mentioned this issue Dec 16, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
performance regression Windows Windows-specific
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Revert "multi: implement wait using winsock events" curl/curl
2 participants
@bagder @tmkk