On 03/20/2018 04:22 PM, Ben Greear wrote:
> On 03/20/2018 04:14 PM, Daniel Stenberg wrote:
>> On Tue, 20 Mar 2018, Ben Greear wrote:
>>
>>>> This new logic updates the rate limit "window" to be no shorter than the
>>>> last three seconds and only updating the timestamps for this when
>>>> switching between the states TOOFAST/PERFORM.
>>>
>>> As far as I can tell, this patch is broken. I am downloading a 20MB file using ftp, with a program compiled against lib-curl. With this patch, download
>>> spikes to 150Mbps (or probably higher, depending on resolution), and then has a long pause for the next URL (that pause might be caused by my own rate-ctrl
>>> across URLs).
>>
>> Yeah, this fix basically reversed the problem and https://github.com/curl/curl/issues/2406 was subsequently filed, although phrased more like a discusssion.
>>
>> I think I need to backpedal a bit and make the period to check be something like 500 milliseconds or something. At least not longer than a second I think.
>>
>>> And/or, what speed showed the previous problem you are trying to fix?
>>
>> The previous problem was that for high speed transfers on for example localhost, it could do the second check the same millisecond as the previous which made
>> the delta time to be zero, thus not limiting it at all.
>>
>> Zero was wrong and 3000 milliseconds is wrong in the other direction...
>
> So, how about 1ms? That will likely be precise enough, and you should be able to
> do the math such that rounding errors are not too bad. 500ms is a very long time
> in the realm of 1 and 10Gbps networks....
>
> I didn't actually look at the math in your rate-limiting code, but I've been using 1ms
> time intervals for such stuff for years and it works well for me...

I guess I should share the algorithm I use, it is basically:

calculate diff between last time I checked and now.
If < 1ms, return

Calculate the amount of bits that should be allowed over the last time interval.
Add those to the 'bucket'.

When it is time to determine if rate is too high or not, check if there are enough
bits in the bucket. If there are, then proceed, else wait until there are enough bits
gathered up.

To decrease bursts after an idle time, you have to limit the bucket to a reasonable amount (like 10-15ms, or at least
the time it takes to send/receive 1 or two of your 'data units'.

Thanks,
Ben

>
> Thanks,
> Ben
>

-- 
Ben Greear <greearb_at_candelatech.com>
Candela Technologies Inc  http://www.candelatech.com
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html