cURL / Mailing Lists / curl-users / Single Mail

curl-users

RE: Throttling Curl

From: Rich Gray <Rich.Gray_at_PlusTechnologies.com>
Date: Wed, 29 May 2002 09:20:56 -0400

Daniel wrote:
>
> Ok, I figure the basics for this would be a function that does:
>
> MAX [ bytes/sec ]
> TIME [ elapsed time in seconds ]
> SIZE [ total number of bytes transferred so far ]
>
> if( SIZE > (MAX * TIME)) {
>
> WAIT ( SIZE/MAX - TIME )
>
> }
>
> This will give a "statistical" maximum through put of MAX. I
> mean, it'll of
> course read and write bursts faster than the given MAX, but
> over time it'll
> average in MAX.

The biggest problem with this algorithm, is that if the writes
get stalled for any period of time, it will try to make up for
lost headway by running wide open until it is back on the bytes
per second slope which originates from the start of the run.

Something with a shorter timeframe (sliding window) might be
better. Below is an off the top of my head, rough, untested
example of how one might hack an inner write function to
literally limit the release of data to some number of bytes
per second. (Please excuse my mutant indenting style.)

    int bytes_per_sec; /* set by parameter */

write_fn(int fd, char *buf, int len)
{
    static time_t now = 0; /* second bytes released in */
    static int left; /* bytes can release this sec. */
    int to_write, written;

    while (len)
       {
       if (bytes_per_second)
          {
          if (now !=time(0))
             {
             left = bytes_per_second;
             now = time(0);
             }
          if (len > left)
             to_write = left;
          else
             to_write = len;
          left -= to_write;
          if (!to_write)
             {
             sleep(1);
             continue;
             }
          }
       else
          to_write = len; /* unthrottled */

       written = write(fd, buf, to_write);
       if (written == -1)
          DEAL WITH ERRORS;
       else
          {
          len -= written;
          buf += written;
          }
       }
   return(WHATEVER);
}

This has one second granularity. One could get even
finer resolution with select, if one wanted.

One other thing (which I've never messed with) might be
to reduce the size of the tcp transmit buffer to, say,
the size one intends to release in a given second.
That would avoid carefully throttling into a large
buffer in case of a connection stall. (When the stall
goes away, all the buffered data is going to go at
full speed.)

Cheers!
Rich

_______________________________________________________________

Don't miss the 2002 Sprint PCS Application Developer's Conference
August 25-28 in Las Vegas -- http://devcon.sprintpcs.com/adp/index.cfm
Received on 2002-05-29