libcurl version: 7.52.1-5+deb9u6

My applications goal: Use pause/unpause in libcurl to ensure a slow client can push back to a fast server so that it slows down while also keeping buffered data in curl for the transfer to a minimum. I want to use the culr_multi_interface so that my application can decide when to transfer data and work with many transfers at a time.

Operations performed:
My application calls curl_multi_perform on a multi-handle that has a single easy-handle ‘add’ed to it. The WRITEFUNCTION gets called to consume data being transferred for a HTTPS GET request. The write function returns CURL_WRITEFUNC_PAUSE to request pause. As data is consumed, in another thread, internal data structures are updated and the curl processing thread calls curl_easy_unpause to unpause the transfer on an easy handle that is already added to the only multi handle being used by the thread. The curl processing thread also calls curl_multi_perform immediately afterwards. Both curl_easy_pause and curl_multi_perform transfer data.

Behavior observed:
Libcurl stops calling the WRITEFUNCTION after pause is requested. However, libcurl keeps receiving data on the connection and buffers it. At some point, when transferring 1GB objects, memory allocation fails and libcurl returns CURLE_OUT_OF_MEMORY.

The above behavior is seen only when using HTTP2.
When using HTTP1.1 to talk to the server, the data is received as transfer is resumed after pauses.

My guess is that the buffered data can get quite large and memory allocation fails. It also defeats the attempt to control flow of data and resource consumption at the client.

Questions:
1. The code in lib/transfer.c:Curl_readwrite does seem to avoid reading from connection when KEEP_RECV_PAUSE is set. Is the intent of the pause functionality to stop reading from socket when WRITING is paused?
2. Is pausing/un-pausing the right way for applications to control the flow of data between libcurl and http server? Is there any other mechanism?
3. My understanding is that curl_multi_perform ultimately calls lib/transfer.c:readwrite_data to read data off the connection from http server and delivers to WRITEFUNCTION using Curl_client_write. When WRITEFUNC requests pause, this state is set in the easy-handle but readwrite_data does not check this state and continues to read data from socket but cannot deliver to the WRITEFUNC. If this is indeed a bug, should readwrite_data check the KEEP_RECV_PAUSE flag in the easy handle and stop reading?
4. Curl_easy_pause may call the writefunc hander synchronously. This goes against the intent of using multi-interface where curl_multi_perform is used to move all data. When using multi interface, is curl_easy_unpause the right way to un-pause paused transfers?
5. Why does maxloops=100 in readwrite_data not prevent it from stopping at reading 100 times from the connection. I sometimes see more than a 1000 invocations of debugfunction that indicate there were 1000 reads of data from the connection. Does the 100 reads somehow transform to 1000+ invocation of debug function? Note that the content I am reading is not compressed.
6. Does this have something to do with the way curl interfaces with nghttp2 for http2?

------------

A change(lib/transfer.c:readwrite_data(…)) to stop reading from a connection when the WRITEFUNCTION has requested pause is:

diff --git a/lib/transfer.c b/lib/transfer.c
index b73f94d9e..d9ef1e394 100644
--- a/lib/transfer.c
+++ b/lib/transfer.c
@@ -844,6 +844,11 @@ static CURLcode readwrite_data(struct Curl_easy *data,
       k->keepon &= ~KEEP_RECV;
     }

+ if(k->keepon & KEEP_RECV_PAUSE) {
+ /* this is a paused transfer */
+ break;
+ }
+
   } while(data_pending(conn) && maxloops--);

   if(maxloops <= 0) {
------------------------------------------

I understand this is against the limit of maxloops=100.
Are there other gotchas with this approach or is this against the principle of what is being done in this code path?

-Dheeraj Sangamkar

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2018-11-03