Re: ratelimits revisited

From: Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>
Date: Thu, 20 Nov 2025 20:45:53 +0000

> 1. How/with what do you takes the measurements?

I have a set of download throttle tests, which is a part of our libcurl readiness tests, which we use when we upgrade libcurl.
The throttle test sets a rate limit via CURLOPT_MAX_RECV_SPEED_LARGE, performs the transfer, and calculates the actual download speed
 (dividing the download size by the measured transfer time).

And typically, I do 1MB download for 8mbps and 16mbps rate limits which I run over networks with ~700 mbps, 50 mbps and 20 mbps network speeds.
This is usually enough to verify that throttling works as expected, and 8.6.0 was very precise in my tests.

> 2. It looks as if you transfer a file with 1 MB size. This I would expect to show fluctuations as the TCP window is not under full control by curl. What do you see with larger downloads?

I tried 5 MB and 20 MB downloads, and the accuracy improved for higher rate limits but still was quite off for 5 MB download.
For 20 MB download it was pretty good.

And comparing with very precise 8.6.0, the actual speeds in my measurements were always higher than the rate limit - a little higher in the best cases and too much higher in the worst cases.
In general, the rate limit is supposed to limit speed and save bandwidth, so it would be great if the actual speed deviations from the rate limit had tendency to deviate below the limit rather than above it.

Here are the results:

########################################################################
# 700 mbps
Speed limit test [LAN (~700 mbps)], iterations=5

Url=http://192.168.128.3/Data/file_5M.bin, max_speed=8000000 bps
time=5008 ms, dnld=5242880 B, speed=8373926 bps, spd_diff=373926 bps, pct=4.7 %
time=5024 ms, dnld=5242880 B, speed=8346903 bps, spd_diff=346903 bps, pct=4.3 %
time=5030 ms, dnld=5242880 B, speed=8337924 bps, spd_diff=337924 bps, pct=4.2 %
time=5027 ms, dnld=5242880 B, speed=8342968 bps, spd_diff=342968 bps, pct=4.3 %
time=5027 ms, dnld=5242880 B, speed=8342356 bps, spd_diff=342356 bps, pct=4.3 %
--------------------------------------------------------------------
avg_deviation=4.4 %, max_deviation=4.7 %

Url=http://192.168.128.3/Data/file_5M.bin, max_speed=16000000 bps
time=2027 ms, dnld=5242880 B, speed=20683513 bps, spd_diff=4683513 bps, pct=29.3 %
time=2023 ms, dnld=5242880 B, speed=20724288 bps, spd_diff=4724288 bps, pct=29.5 %
time=2022 ms, dnld=5242880 B, speed=20740076 bps, spd_diff=4740076 bps, pct=29.6 %
time=2013 ms, dnld=5242880 B, speed=20835609 bps, spd_diff=4835609 bps, pct=30.2 %
time=2008 ms, dnld=5242880 B, speed=20879650 bps, spd_diff=4879650 bps, pct=30.5 %
--------------------------------------------------------------------
avg_deviation=29.8 %, max_deviation=30.5 %

Url=http://192.168.128.3/Data/file_5M.bin, max_speed=32000000 bps
time=1020 ms, dnld=5242880 B, speed=41085426 bps, spd_diff=9085426 bps, pct=28.4 %
time=1019 ms, dnld=5242880 B, speed=41141105 bps, spd_diff=9141105 bps, pct=28.6 %
time=1025 ms, dnld=5242880 B, speed=40919404 bps, spd_diff=8919404 bps, pct=27.9 %
time=1020 ms, dnld=5242880 B, speed=41104532 bps, spd_diff=9104532 bps, pct=28.5 %
time=1019 ms, dnld=5242880 B, speed=41124885 bps, spd_diff=9124885 bps, pct=28.5 %
--------------------------------------------------------------------
avg_deviation=28.4 %, max_deviation=28.6 %

--------------------------------------------------------------------
Speed limit test [LAN (~700 mbps)], iterations=5

Url=http://192.168.128.3/Data/file_20M.bin, max_speed=8000000 bps
time=20044 ms, dnld=20971520 B, speed=8369864 bps, spd_diff=369864 bps, pct=4.6 %
time=20030 ms, dnld=20971520 B, speed=8375842 bps, spd_diff=375842 bps, pct=4.7 %
time=20031 ms, dnld=20971520 B, speed=8375461 bps, spd_diff=375461 bps, pct=4.7 %
time=20018 ms, dnld=20971520 B, speed=8380855 bps, spd_diff=380855 bps, pct=4.8 %
time=20014 ms, dnld=20971520 B, speed=8382437 bps, spd_diff=382437 bps, pct=4.8 %
--------------------------------------------------------------------
avg_deviation=4.7 %, max_deviation=4.8 %

Url=http://192.168.128.3/Data/file_20M.bin, max_speed=16000000 bps
time=10030 ms, dnld=20971520 B, speed=16726238 bps, spd_diff=726238 bps, pct=4.5 %
time=10031 ms, dnld=20971520 B, speed=16724039 bps, spd_diff=724039 bps, pct=4.5 %
time=10028 ms, dnld=20971520 B, speed=16730169 bps, spd_diff=730169 bps, pct=4.6 %
time=10030 ms, dnld=20971520 B, speed=16726868 bps, spd_diff=726868 bps, pct=4.5 %
time=10022 ms, dnld=20971520 B, speed=16738937 bps, spd_diff=738937 bps, pct=4.6 %
--------------------------------------------------------------------
avg_deviation=4.6 %, max_deviation=4.6 %

Url=http://192.168.128.3/Data/file_20M.bin, max_speed=32000000 bps
time=5020 ms, dnld=20971520 B, speed=33420326 bps, spd_diff=1420326 bps, pct=4.4 %
time=5023 ms, dnld=20971520 B, speed=33400677 bps, spd_diff=1400677 bps, pct=4.4 %
time=5031 ms, dnld=20971520 B, speed=33343534 bps, spd_diff=1343534 bps, pct=4.2 %
time=5007 ms, dnld=20971520 B, speed=33501246 bps, spd_diff=1501246 bps, pct=4.7 %
time=5025 ms, dnld=20971520 B, speed=33384424 bps, spd_diff=1384424 bps, pct=4.3 %
--------------------------------------------------------------------
avg_deviation=4.4 %, max_deviation=4.7 %

########################################################################
# 50 mbps
Speed limit test [50 mbps], iterations=5

Url=http://192.168.128.3/Data/file_5M.bin, max_speed=8000000 bps
time=5005 ms, dnld=5242880 B, speed=8378652 bps, spd_diff=378652 bps, pct=4.7 %
time=5028 ms, dnld=5242880 B, speed=8340357 bps, spd_diff=340357 bps, pct=4.3 %
time=5024 ms, dnld=5242880 B, speed=8347930 bps, spd_diff=347930 bps, pct=4.3 %
time=5025 ms, dnld=5242880 B, speed=8346587 bps, spd_diff=346587 bps, pct=4.3 %
time=5030 ms, dnld=5242880 B, speed=8337377 bps, spd_diff=337377 bps, pct=4.2 %
--------------------------------------------------------------------
avg_deviation=4.4 %, max_deviation=4.7 %

Url=http://192.168.128.3/Data/file_5M.bin, max_speed=16000000 bps
time=2183 ms, dnld=5242880 B, speed=19212034 bps, spd_diff=3212034 bps, pct=20.1 %
time=2160 ms, dnld=5242880 B, speed=19418070 bps, spd_diff=3418070 bps, pct=21.4 %
time=2165 ms, dnld=5242880 B, speed=19371115 bps, spd_diff=3371115 bps, pct=21.1 %
time=2160 ms, dnld=5242880 B, speed=19411031 bps, spd_diff=3411031 bps, pct=21.3 %
time=2166 ms, dnld=5242880 B, speed=19356346 bps, spd_diff=3356346 bps, pct=21.0 %
--------------------------------------------------------------------
avg_deviation=21.0 %, max_deviation=21.4 %

Url=http://192.168.128.3/Data/file_5M.bin, max_speed=32000000 bps
time=1178 ms, dnld=5242880 B, speed=35576232 bps, spd_diff=3576232 bps, pct=11.2 %
time=1172 ms, dnld=5242880 B, speed=35763704 bps, spd_diff=3763704 bps, pct=11.8 %
time=1173 ms, dnld=5242880 B, speed=35747787 bps, spd_diff=3747787 bps, pct=11.7 %
time=1153 ms, dnld=5242880 B, speed=36361234 bps, spd_diff=4361234 bps, pct=13.6 %
time=1168 ms, dnld=5242880 B, speed=35897606 bps, spd_diff=3897606 bps, pct=12.2 %
--------------------------------------------------------------------
avg_deviation=12.1 %, max_deviation=13.6 %

Speed limit test [50 mbps], iterations=5

Url=http://192.168.128.3/Data/file_20M.bin, max_speed=8000000 bps
time=20125 ms, dnld=20971520 B, speed=8336446 bps, spd_diff=336446 bps, pct=4.2 %
time=20120 ms, dnld=20971520 B, speed=8338231 bps, spd_diff=338231 bps, pct=4.2 %
time=20111 ms, dnld=20971520 B, speed=8341948 bps, spd_diff=341948 bps, pct=4.3 %
time=20128 ms, dnld=20971520 B, speed=8334939 bps, spd_diff=334939 bps, pct=4.2 %
time=20127 ms, dnld=20971520 B, speed=8335310 bps, spd_diff=335310 bps, pct=4.2 %
--------------------------------------------------------------------
avg_deviation=4.2 %, max_deviation=4.3 %

Url=http://192.168.128.3/Data/file_20M.bin, max_speed=16000000 bps
time=10144 ms, dnld=20971520 B, speed=16537472 bps, spd_diff=537472 bps, pct=3.4 %
time=10119 ms, dnld=20971520 B, speed=16578557 bps, spd_diff=578557 bps, pct=3.6 %
time=10127 ms, dnld=20971520 B, speed=16565747 bps, spd_diff=565747 bps, pct=3.5 %
time=10125 ms, dnld=20971520 B, speed=16569468 bps, spd_diff=569468 bps, pct=3.6 %
time=10126 ms, dnld=20971520 B, speed=16566955 bps, spd_diff=566955 bps, pct=3.5 %
--------------------------------------------------------------------
avg_deviation=3.5 %, max_deviation=3.6 %

Url=http://192.168.128.3/Data/file_20M.bin, max_speed=32000000 bps
time=5126 ms, dnld=20971520 B, speed=32728186 bps, spd_diff=728186 bps, pct=2.3 %
time=5110 ms, dnld=20971520 B, speed=32826439 bps, spd_diff=826439 bps, pct=2.6 %
time=5116 ms, dnld=20971520 B, speed=32791890 bps, spd_diff=791890 bps, pct=2.5 %
time=5144 ms, dnld=20971520 B, speed=32614811 bps, spd_diff=614811 bps, pct=1.9 %
time=5110 ms, dnld=20971520 B, speed=32832108 bps, spd_diff=832108 bps, pct=2.6 %
--------------------------------------------------------------------
avg_deviation=2.4 %, max_deviation=2.6 %

########################################################################
# 20 mbps
Speed limit test [20 mbps], iterations=5

Url=http://192.168.128.3/Data/file_5M.bin, max_speed=8000000 bps
time=5020 ms, dnld=5242880 B, speed=8354037 bps, spd_diff=354037 bps, pct=4.4 %
time=5048 ms, dnld=5242880 B, speed=8308568 bps, spd_diff=308568 bps, pct=3.9 %
time=5033 ms, dnld=5242880 B, speed=8333496 bps, spd_diff=333496 bps, pct=4.2 %
time=5046 ms, dnld=5242880 B, speed=8311638 bps, spd_diff=311638 bps, pct=3.9 %
time=5041 ms, dnld=5242880 B, speed=8319221 bps, spd_diff=319221 bps, pct=4.0 %
--------------------------------------------------------------------
avg_deviation=4.1 %, max_deviation=4.4 %

Url=http://192.168.128.3/Data/file_5M.bin, max_speed=16000000 bps
time=2455 ms, dnld=5242880 B, speed=17082607 bps, spd_diff=1082607 bps, pct=6.8 %
time=2433 ms, dnld=5242880 B, speed=17233945 bps, spd_diff=1233945 bps, pct=7.7 %
time=2436 ms, dnld=5242880 B, speed=17211464 bps, spd_diff=1211464 bps, pct=7.6 %
time=2435 ms, dnld=5242880 B, speed=17219962 bps, spd_diff=1219962 bps, pct=7.6 %
time=2485 ms, dnld=5242880 B, speed=16874623 bps, spd_diff=874623 bps, pct=5.5 %
--------------------------------------------------------------------
avg_deviation=7.0 %, max_deviation=7.7 %

To summarize, I think the results show that rate limiting works more or less OK for large downloads (i.e. 20 MB),
get worse for smaller downloads (i.e. 5MB), and deteriorates significantly for some rate limits for small downloads like 1MB (where it just doesn't work for some network speeds and rate limits).

Let me know if you want me to run more tests with different network speeds, rate limits and download sizes.

Thanks,
Dmitry


-----Original Message-----
From: Stefan Eissing <stefan_at_eissing.org>
Sent: Thursday, November 20, 2025 12:01 AM
To: Dmitry Karpov <dkarpov_at_roku.com>
Cc: libcurl development <curl-library_at_lists.haxx.se>
Subject: [EXTERNAL] Re: ratelimits revisited



> Am 20.11.2025 um 02:04 schrieb Dmitry Karpov <dkarpov_at_roku.com>:
>
> Hi Stefan,
>
> Thanks a lot for your effort to make rate limiting better!
> I tried your PR on my speed throttling tests, and unfortunately it doesn't work well on different network speeds and speed limits.

Thanks for testing this. 2 questions
1. How/with what do you takes the measurements?
2. It looks as if you transfer a file with 1 MB size. This I would expect to show fluctuations as the TCP window is not under full control by curl. What do you see with larger downloads?

- Stefan

>
> Here are the stats from my tests (HTTP1):
>
> Speed limit test [LAN (~700 mbps)], iterations=10
>
> Url=http://192.168.1.218/Data/file_1M.bin, max_speed=8000000 bps
> time=1019 ms, dnld=1048576 B, speed=8230254 bps, spd_diff=230254 bps,
> pct=2.9 %
> time=1029 ms, dnld=1048576 B, speed=8144532 bps, spd_diff=144532 bps,
> pct=1.8 % ...
> time=1017 ms, dnld=1048576 B, speed=8246162 bps, spd_diff=246162 bps,
> pct=3.1 %
> time=1029 ms, dnld=1048576 B, speed=8148398 bps, spd_diff=148398 bps,
> pct=1.9 %
> time=1033 ms, dnld=1048576 B, speed=8119992 bps, spd_diff=119992 bps,
> pct=1.5 %
> --------------------------------------------------------------------
> avg_deviation=2.2 %, max_deviation=3.1 %
>
> Url=http://192.168.1.218/Data/file_1M.bin, max_speed=16000000 bps
> time=63 ms, dnld=1048576 B, speed=132741430 bps, spd_diff=116741430
> bps, pct=729.6 %
> time=63 ms, dnld=1048576 B, speed=132284678 bps, spd_diff=116284678
> bps, pct=726.8 %
> time=20 ms, dnld=1048576 B, speed=408757735 bps, spd_diff=392757735
> bps, pct=2454.7 %
> time=45 ms, dnld=1048576 B, speed=186236793 bps, spd_diff=170236793
> bps, pct=1064.0 %
> time=54 ms, dnld=1048576 B, speed=154633279 bps, spd_diff=138633279
> bps, pct=866.5 %
> time=17 ms, dnld=1048576 B, speed=482284071 bps, spd_diff=466284071
> bps, pct=2914.3 %
> time=18 ms, dnld=1048576 B, speed=461120284 bps, spd_diff=445120284
> bps, pct=2782.0 %
> time=18 ms, dnld=1048576 B, speed=450918004 bps, spd_diff=434918004
> bps, pct=2718.2 %
> time=19 ms, dnld=1048576 B, speed=426558187 bps, spd_diff=410558187
> bps, pct=2566.0 %
> time=16 ms, dnld=1048576 B, speed=497665980 bps, spd_diff=481665980
> bps, pct=3010.4 %
> --------------------------------------------------------------------
> avg_deviation=1983.3 %, max_deviation=3010.4 %
>
>
> Speed limit test [50 mbps], iterations=10
>
> Url=http://192.168.1.218/Data/file_1M.bin, max_speed=8000000 bps
> time=1025 ms, dnld=1048576 B, speed=8179852 bps, spd_diff=179852 bps,
> pct=2.2 %
> time=1018 ms, dnld=1048576 B, speed=8236288 bps, spd_diff=236288 bps,
> pct=3.0 % ...
> time=1018 ms, dnld=1048576 B, speed=8234946 bps, spd_diff=234946 bps,
> pct=2.9 %
> time=1033 ms, dnld=1048576 B, speed=8113928 bps, spd_diff=113928 bps,
> pct=1.4 %
> time=1010 ms, dnld=1048576 B, speed=8300803 bps, spd_diff=300803 bps,
> pct=3.8 %
> --------------------------------------------------------------------
> avg_deviation=2.6 %, max_deviation=3.8 %
>
> Url=http://192.168.1.218/Data/file_1M.bin, max_speed=16000000 bps
> time=191 ms, dnld=1048576 B, speed=43763514 bps, spd_diff=27763514
> bps, pct=173.5 %
> time=174 ms, dnld=1048576 B, speed=48007792 bps, spd_diff=32007792
> bps, pct=200.0 % ...
> time=191 ms, dnld=1048576 B, speed=43896683 bps, spd_diff=27896683
> bps, pct=174.4 %
> time=188 ms, dnld=1048576 B, speed=44474412 bps, spd_diff=28474412
> bps, pct=178.0 %
> --------------------------------------------------------------------
> avg_deviation=180.4 %, max_deviation=200.0 %
>
>
> Speed limit test [20 mbps], iterations=10
>
> Url=http://192.168.1.218/Data/file_1M.bin, max_speed=8000000 bps
> time=1018 ms, dnld=1048576 B, speed=8234509 bps, spd_diff=234509 bps,
> pct=2.9 %
> time=1030 ms, dnld=1048576 B, speed=8144106 bps, spd_diff=144106 bps,
> pct=1.8 % ...
> time=1019 ms, dnld=1048576 B, speed=8226810 bps, spd_diff=226810 bps,
> pct=2.8 %
> time=1021 ms, dnld=1048576 B, speed=8209510 bps, spd_diff=209510 bps,
> pct=2.6 %
> time=1029 ms, dnld=1048576 B, speed=8146626 bps, spd_diff=146626 bps,
> pct=1.8 %
> --------------------------------------------------------------------
> avg_deviation=2.1 %, max_deviation=2.9 %
>
> Url=http://192.168.1.218/Data/file_1M.bin, max_speed=16000000 bps
> time=455 ms, dnld=1048576 B, speed=18427202 bps, spd_diff=2427202 bps,
> pct=15.2 %
> time=449 ms, dnld=1048576 B, speed=18665799 bps, spd_diff=2665799 bps,
> pct=16.7 % ...
> time=455 ms, dnld=1048576 B, speed=18429862 bps, spd_diff=2429862 bps,
> pct=15.2 %
> time=471 ms, dnld=1048576 B, speed=17795616 bps, spd_diff=1795616 bps,
> pct=11.2 %
> --------------------------------------------------------------------
> avg_deviation=16.3 %, max_deviation=20.6 %
>
> By some reason, the rate limiting worked fine for 8 kbps limit, but was totally off for 16 kbps.
>
> Thanks,
> Dmitry Karpov
>
>
>
> -----Original Message-----
> From: curl-library <curl-library-bounces_at_lists.haxx.se> On Behalf Of
> Stefan Eissing via curl-library
> Sent: Wednesday, November 19, 2025 4:26 AM
> To: curl-library_at_lists.haxx.se
> Cc: Stefan Eissing <stefan_at_eissing.org>
> Subject: [EXTERNAL] ratelimits revisited
>
> Some people recently expressed interest in a more precise implementation of --limit-rate and I had a stab at it the other day. Please see:
>
> https://github.com/curl/curl/pull/19384
>
> - increased precision on all HTTP protocols
> - improved stream window handling on HTTP/2 transfers
> - implementation of rate limiting in HTTP/3 transfers with ngtcp2
>
> The increased precision should not come at extra cpu costs. If you would take that out for a spin in your environment, I'd be happy to get some feedback on this change.
>
> Kind Regards,
> Stefan
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2025-11-20