On real time applications it will be ideal if we could make sure
transfers go smoothly and processing proportionate chunks of data.

So I did try to use CURLOPT_MAX_RECV_SPEED_LARGE to limit data transfers
using a 1KB buffer to 30KB/s and calling curl_multi_perform at 30 Hz.

     curl_easy_setopt(curl_easy_handle, CURLOPT_BUFFERSIZE, 1024);
     curl_easy_setopt(curl_easy_handle, CURLOPT_MAX_RECV_SPEED_LARGE,
(curl_off_t) 30720);

I also have a call back write function and pass a pointer to the
implementing class instance as the user data for the callback.

     curl_easy_setopt(curl_easy_handle, CURLOPT_WRITEFUNCTION,
curl_write_function);
     curl_easy_setopt(curl_easy_handle, CURLOPT_WRITEDATA, this);

I was expecting the write callback to be called more or less once per
curl_multi_perform with a payload of 1K per frame. But what I see is a
bunch of many 1K callbacks until it reaches 30 for 1 second, then
nothing then it resumes.

So I noticed that CURLOPT_MAX_RECV_SPEED_LARGE only restricts calling
Curl_readwrite if the current download speed exceeds the max_recv_speed
if it's set, by switching between CURLM_STATE_PERFORM and
CURLM_STATE_TOOFAST states in lib\multi.c function multi_runsingle

If it's within the transfer up/download speeds it will call
Curl_readwrite (implemented in lib\transfer.c)

Curl_readwrite does then a select to see if there is data ready for send
or receive.

The sending part is fine: It calls readwrite_upload and that method only
sends one buffer's worth via Curl_write because it's using a do while
(0) loop.

The receiving part is where I see a bottleneck for smooth real-time
transfers when it calls readwrite_data (implemented in lib\transfer.c as
well)

This is implemented with a do {} while(data_pending(conn)) loop meaning
it's going to call Curl_read() as many times as needed until there is no
more data. This will break any speed limits set by
CURLOPT_MAX_RECV_SPEED_LARGE. The callback set with
CURLOPT_WRITEFUNCTION will be called many times within this loop until
no more data is available.

I'm suggesting adding a Curl_pgrsUpdate() to update the download speed,
then checking if we are over our speed and if so break the loop.

So readwrite_data() 's do loop will look like this at the end of the
loop code:

     if(is_empty_data) {
       /* if we received nothing, the server closed the connection and we
          are done */
       k->keepon &= ~KEEP_RECV;
     }
     /*
      * Suggested fix
      */
     /* update connection progress so we can check if over recv speed */
     Curl_pgrsUpdate(conn);
     /* check if over recv speed */
     if((data->set.max_recv_speed > 0) &&
        (data->progress.dlspeed > data->set.max_recv_speed)) {
         break; /* get out of loop */
     }
     /*
      * End Suggested fix
      */
} while(data_pending(conn));

This will only break the loop if max_recv_speed was set with
CURLOPT_MAX_RECV_SPEED_LARGE and the current download speed exceeds it.

This change will have the following benefits:
1) it will maintain the speed limit even from the first data frame.
2) CURLOPT_WRITEFUNCTION callback will be called mover evenly (instead
of burst at a time when CURLOPT_MAX_RECV_SPEED_LARGE is used)
3) It will call the progress callback more often giving more granularity
of data being processed regardless of CURLOPT_MAX_RECV_SPEED_LARGE being
set or not.
4) real-time friendly for devices and applications that need a tight
control over bandwidth and resources.
5) it will handle very low speeds too balancing processing and transfer
times evenly.

Downside: it will call progress update many more times per
curl_multi_perform depending on the read buffersize (not sure if a
downside but extra function calls could introduce overhead depending
what the progress callback function does).

Thanks,

Miguel

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2012-08-03