Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
Re: ratelimits revisited
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Stefan Eissing via curl-library <curl-library_at_lists.haxx.se>
Date: Sat, 22 Nov 2025 11:59:32 +0100
> Am 21.11.2025 um 20:42 schrieb Dmitry Karpov <dkarpov_at_roku.com>:
>
>> How to you measure the tranfer time exactly? I'd need to reproduce your measurements to drive this forward.
>
> I use steady clock for that (provided by std::chrono::steady_clock), and I take time points at transfer start and transfer end.
> The (end - start) duration is my measured transfer time as client sees it.
>
> But you can get very similar results if you measure transfer time using curl_easy_getinfo and CURLINFO_TOTAL_TIME_T.
> I tried it, and the results are almost identical to my measurements with steady clock.
Thanks for the explanation. That makes sense then. The PR had taken the approach of "download n bytes per second", meaning at the start of each second interval, curl received up to n bytes. This results in the following:
5MB download with 2 MB/s:
+0----+1----+2----+3----
2MB 4MB 5MB
with an overall time then of sth like 2.1 seconds and an average of 2.4 MB/s. *But while it was ongoing* the 2MB/s was exactly obeyed.
The last chunk of the download never takes the full second interval, increasing the total rate, although it stayed at 2MB/s all the time. The shorter the download is, the more this increases the total average.
I updated the PR to go down to 500ms intervals (if the rate is >= 32kb/s). This makes the last incomplete interval smaller, giving more precision.
We can make those intervals even smaller, increasing the precision more and more, but then CPU usage will increase because there are more timeouts happening. This is a tradeoff. We do not want to receive, in exterme, a single byte per microsecond for ultimate precision. There seems to be little value in that. Tradeoffs.
Another, silly strategy would be to hold the transfer at the end until the full second has passed. That also would give precise speed numbers, but is rather futile.
Cheers,
Stefan
>
> Thanks,
> Dmitry
>
>
> -----Original Message-----
> From: Stefan Eissing <stefan_at_eissing.org>
> Sent: Friday, November 21, 2025 12:25 AM
> To: Dmitry Karpov <dkarpov_at_roku.com>
> Cc: libcurl development <curl-library_at_lists.haxx.se>
> Subject: [EXTERNAL] Re: ratelimits revisited
>
>
>
>> Am 20.11.2025 um 21:45 schrieb Dmitry Karpov <dkarpov_at_roku.com>:
>>
>>> 1. How/with what do you takes the measurements?
>>
>> I have a set of download throttle tests, which is a part of our libcurl readiness tests, which we use when we upgrade libcurl.
>> The throttle test sets a rate limit via CURLOPT_MAX_RECV_SPEED_LARGE,
>> performs the transfer, and calculates the actual download speed (dividing the download size by the measured transfer time).
>
> How to you measure the tranfer time exactly? I'd need to reproduce your measurements to drive this forward.
>
> Thanks,
> Stefan
Date: Sat, 22 Nov 2025 11:59:32 +0100
> Am 21.11.2025 um 20:42 schrieb Dmitry Karpov <dkarpov_at_roku.com>:
>
>> How to you measure the tranfer time exactly? I'd need to reproduce your measurements to drive this forward.
>
> I use steady clock for that (provided by std::chrono::steady_clock), and I take time points at transfer start and transfer end.
> The (end - start) duration is my measured transfer time as client sees it.
>
> But you can get very similar results if you measure transfer time using curl_easy_getinfo and CURLINFO_TOTAL_TIME_T.
> I tried it, and the results are almost identical to my measurements with steady clock.
Thanks for the explanation. That makes sense then. The PR had taken the approach of "download n bytes per second", meaning at the start of each second interval, curl received up to n bytes. This results in the following:
5MB download with 2 MB/s:
+0----+1----+2----+3----
2MB 4MB 5MB
with an overall time then of sth like 2.1 seconds and an average of 2.4 MB/s. *But while it was ongoing* the 2MB/s was exactly obeyed.
The last chunk of the download never takes the full second interval, increasing the total rate, although it stayed at 2MB/s all the time. The shorter the download is, the more this increases the total average.
I updated the PR to go down to 500ms intervals (if the rate is >= 32kb/s). This makes the last incomplete interval smaller, giving more precision.
We can make those intervals even smaller, increasing the precision more and more, but then CPU usage will increase because there are more timeouts happening. This is a tradeoff. We do not want to receive, in exterme, a single byte per microsecond for ultimate precision. There seems to be little value in that. Tradeoffs.
Another, silly strategy would be to hold the transfer at the end until the full second has passed. That also would give precise speed numbers, but is rather futile.
Cheers,
Stefan
>
> Thanks,
> Dmitry
>
>
> -----Original Message-----
> From: Stefan Eissing <stefan_at_eissing.org>
> Sent: Friday, November 21, 2025 12:25 AM
> To: Dmitry Karpov <dkarpov_at_roku.com>
> Cc: libcurl development <curl-library_at_lists.haxx.se>
> Subject: [EXTERNAL] Re: ratelimits revisited
>
>
>
>> Am 20.11.2025 um 21:45 schrieb Dmitry Karpov <dkarpov_at_roku.com>:
>>
>>> 1. How/with what do you takes the measurements?
>>
>> I have a set of download throttle tests, which is a part of our libcurl readiness tests, which we use when we upgrade libcurl.
>> The throttle test sets a rate limit via CURLOPT_MAX_RECV_SPEED_LARGE,
>> performs the transfer, and calculates the actual download speed (dividing the download size by the measured transfer time).
>
> How to you measure the tranfer time exactly? I'd need to reproduce your measurements to drive this forward.
>
> Thanks,
> Stefan
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2025-11-22