Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
Re: CPU usage since 8.7.1
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Stefan Eissing via curl-library <curl-library_at_lists.haxx.se>
Date: Thu, 30 Oct 2025 10:56:20 +0100
> Am 29.10.2025 um 22:12 schrieb Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>:
>
> I ran the same throttling tests on 8.16.0 that helped to find the speed throttling issue on 8.7, and I can tell that the throttling accuracy in 8.16.0 is significantly worse than in 8.6.0 and 8.7.1.
>
> My tests use 1MB downloads on network speeds: 700mbps, 50mbps and 20mbps, with 8mbps and 16mbps throttling speeds.
> The test measures the performed transfers speeds on 10 downloads and calculates average and max deviations from the rate limits.
>
> Here is the brief description of test results:
>
> 8.6.0 (and 8.7.1)
> Network speed: 700mbps, Rate limit: 8mbps, Avg deviation: -0.86%, Max deviation: -1.5%.
> Network speed: 700mbps, Rate limit: 16mbps, Avg deviation: -1.72%, Max deviation: -2.8%.
>
> Network speed: 50mbps, Rate limit: 8mbps, Avg deviation: -0.37%, Max deviation: -1%.
> Network speed: 50mbps, Rate limit: 16mbps, Avg deviation: - 1.64%, Max deviation: -3.2%.
>
> Network speed: 20mbps, Rate limit: 8mbps, Avg deviation: -0.6%, Max deviation: -1.1%.
> Network speed: 20mbps, Rate limit: 16mbps, Avg deviation: -1.37%, Max deviation: -1.7%.
>
> 8.16.0
> Network speed: 700mbps, Rate limit: 8mbps, Avg deviation: 4.65%, Max deviation: 13.7%.
> Network speed: 700mbps, Rate limit: 16mbps, Avg deviation: 6.69%, Max deviation: 17.7%.
>
> Network speed: 50mbps, Rate limit: 8mbps, Avg deviation: 7.03%, Max deviation: 10.2%.
> Network speed: 50mbps, Rate limit: 16mbps, Avg deviation: 9.99%, Max deviation: 15%.
>
> Network speed: 20mbps, Rate limit: 8mbps, Avg deviation: 10.89%, Max deviation: 15.5%.
> Network speed: 20mbps, Rate limit: 16mbps, Avg deviation: 5.34%, Max deviation: 14.5%.
>
> As we can see, the 8.6.0 (and 8.7.1) has almost perfect rate limit accuracy with a tendency to curb the download speed a little but below the limit,
> which is a good thing that makes the download speed unlikely to go beyond the limit.
>
> On the other hand, the 8.16.0 has wider deviations from the rate limits with a tendency to use more network bandwidth than the limit specifies.
> And the max deviations above 10% make it difficult to have reliable predictions of the throttled download speeds.
>
> If the almost perfect accuracy of 8.6.0/8.7.1 comes with a price of much higher CPU utilization, which maybe not acceptable for everyone,
> then maybe we need to adjust the rate limiting mechanism and make it to provide just 5% accuracy with a tendency to use less network bandwidth than the rate limit,
> when the precise accuracy is not possible.
>
> The tendency to use more bandwidth in the current mechanism creates problems when rate limits are specifically set to prevent low priority downloads from stealing
> network bandwidth from downloads with higher priority.
We'd have to revisit rate-limited/paused transfers in the future. Your measurements show that this could be better.
Not sure when we'll find the time for that. So, I cannot promise anything.
Cheers,
Stefan
>
> Thanks,
> Dmitry Karpov
>
>
>
>
> -----Original Message-----
> From: curl-library <curl-library-bounces_at_lists.haxx.se> On Behalf Of Dmitry Karpov via curl-library
> Sent: Tuesday, October 28, 2025 11:38 AM
> To: libcurl development <curl-library_at_lists.haxx.se>
> Cc: Dmitry Karpov <dkarpov_at_roku.com>
> Subject: [EXTERNAL] Re: CPU usage since 8.7.1
>
> It was me who reported it back then.
> The precision loss of the rated downloads was more than 10% in some cases, which was above our 5% tolerance.
>
> It caused problems in multi-download prioritized scenarios where rated downloads with less priority were stealing bandwidth from the downloads with higher priority.
>
> I am also concerned about CPU usage, but 10% rate precision deviation is too high.
> And in our case (video streaming application), the problem was that the prioritized video got worse quality and other problems which affected our users.
>
> Ideally, the optimal solution would be to avoid busy loops and using too many CPU cycles, and provide 5% rate accuracy.
>
> Thanks!
> Dmitry Karpov
>
> -----Original Message-----
> From: curl-library <curl-library-bounces_at_lists.haxx.se> On Behalf Of Gleb Smirnoff via curl-library
> Sent: Tuesday, October 28, 2025 11:11 AM
> To: Daniel Stenberg <daniel_at_haxx.se>; Stefan Eissing <stefan_at_eissing.org>
> Cc: Gleb Smirnoff <glebius_at_glebi.us>; Gleb Smirnoff via curl-library <curl-library_at_lists.haxx.se>
> Subject: [EXTERNAL] Re: CPU usage since 8.7.1
>
> On Tue, Oct 28, 2025 at 05:32:27PM +0100, Daniel Stenberg wrote:
> D> On Tue, 28 Oct 2025, Gleb Smirnoff via curl-library wrote:
> D>
> D> > So here I am raising this question. What needs to be done to get
> D> > CPU savings of 8.6.0 level and how can we help here?
> D>
> D> As recall things, the change that introduced the new (current)
> D> behavior was done to enhance the accuracy of the rate limit logic, at
> D> the expense of CPU.
> D>
> D> The change was pushed for by someone who argued that the increased
> D> precision was valuable for them if I remember correctly.
> D>
> D> Can we find a way to have both ways and let the user select which?
> D> Maybe for example by maybe something as silly and simple as doing
> D> CURLOPT_MAX_RECV_SPEED_LARGE % 10000 and if that is non-zero, higher
> D> precision is wanted.
>
> I added Stefan to explicit Cc. Since he very recently added 24badd29f titled "limit-rate revisited" (of course this conlicted with out patch again), he definitely should have an opinion on how this can be solved.
>
> --
> Gleb Smirnoff
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
Date: Thu, 30 Oct 2025 10:56:20 +0100
> Am 29.10.2025 um 22:12 schrieb Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>:
>
> I ran the same throttling tests on 8.16.0 that helped to find the speed throttling issue on 8.7, and I can tell that the throttling accuracy in 8.16.0 is significantly worse than in 8.6.0 and 8.7.1.
>
> My tests use 1MB downloads on network speeds: 700mbps, 50mbps and 20mbps, with 8mbps and 16mbps throttling speeds.
> The test measures the performed transfers speeds on 10 downloads and calculates average and max deviations from the rate limits.
>
> Here is the brief description of test results:
>
> 8.6.0 (and 8.7.1)
> Network speed: 700mbps, Rate limit: 8mbps, Avg deviation: -0.86%, Max deviation: -1.5%.
> Network speed: 700mbps, Rate limit: 16mbps, Avg deviation: -1.72%, Max deviation: -2.8%.
>
> Network speed: 50mbps, Rate limit: 8mbps, Avg deviation: -0.37%, Max deviation: -1%.
> Network speed: 50mbps, Rate limit: 16mbps, Avg deviation: - 1.64%, Max deviation: -3.2%.
>
> Network speed: 20mbps, Rate limit: 8mbps, Avg deviation: -0.6%, Max deviation: -1.1%.
> Network speed: 20mbps, Rate limit: 16mbps, Avg deviation: -1.37%, Max deviation: -1.7%.
>
> 8.16.0
> Network speed: 700mbps, Rate limit: 8mbps, Avg deviation: 4.65%, Max deviation: 13.7%.
> Network speed: 700mbps, Rate limit: 16mbps, Avg deviation: 6.69%, Max deviation: 17.7%.
>
> Network speed: 50mbps, Rate limit: 8mbps, Avg deviation: 7.03%, Max deviation: 10.2%.
> Network speed: 50mbps, Rate limit: 16mbps, Avg deviation: 9.99%, Max deviation: 15%.
>
> Network speed: 20mbps, Rate limit: 8mbps, Avg deviation: 10.89%, Max deviation: 15.5%.
> Network speed: 20mbps, Rate limit: 16mbps, Avg deviation: 5.34%, Max deviation: 14.5%.
>
> As we can see, the 8.6.0 (and 8.7.1) has almost perfect rate limit accuracy with a tendency to curb the download speed a little but below the limit,
> which is a good thing that makes the download speed unlikely to go beyond the limit.
>
> On the other hand, the 8.16.0 has wider deviations from the rate limits with a tendency to use more network bandwidth than the limit specifies.
> And the max deviations above 10% make it difficult to have reliable predictions of the throttled download speeds.
>
> If the almost perfect accuracy of 8.6.0/8.7.1 comes with a price of much higher CPU utilization, which maybe not acceptable for everyone,
> then maybe we need to adjust the rate limiting mechanism and make it to provide just 5% accuracy with a tendency to use less network bandwidth than the rate limit,
> when the precise accuracy is not possible.
>
> The tendency to use more bandwidth in the current mechanism creates problems when rate limits are specifically set to prevent low priority downloads from stealing
> network bandwidth from downloads with higher priority.
We'd have to revisit rate-limited/paused transfers in the future. Your measurements show that this could be better.
Not sure when we'll find the time for that. So, I cannot promise anything.
Cheers,
Stefan
>
> Thanks,
> Dmitry Karpov
>
>
>
>
> -----Original Message-----
> From: curl-library <curl-library-bounces_at_lists.haxx.se> On Behalf Of Dmitry Karpov via curl-library
> Sent: Tuesday, October 28, 2025 11:38 AM
> To: libcurl development <curl-library_at_lists.haxx.se>
> Cc: Dmitry Karpov <dkarpov_at_roku.com>
> Subject: [EXTERNAL] Re: CPU usage since 8.7.1
>
> It was me who reported it back then.
> The precision loss of the rated downloads was more than 10% in some cases, which was above our 5% tolerance.
>
> It caused problems in multi-download prioritized scenarios where rated downloads with less priority were stealing bandwidth from the downloads with higher priority.
>
> I am also concerned about CPU usage, but 10% rate precision deviation is too high.
> And in our case (video streaming application), the problem was that the prioritized video got worse quality and other problems which affected our users.
>
> Ideally, the optimal solution would be to avoid busy loops and using too many CPU cycles, and provide 5% rate accuracy.
>
> Thanks!
> Dmitry Karpov
>
> -----Original Message-----
> From: curl-library <curl-library-bounces_at_lists.haxx.se> On Behalf Of Gleb Smirnoff via curl-library
> Sent: Tuesday, October 28, 2025 11:11 AM
> To: Daniel Stenberg <daniel_at_haxx.se>; Stefan Eissing <stefan_at_eissing.org>
> Cc: Gleb Smirnoff <glebius_at_glebi.us>; Gleb Smirnoff via curl-library <curl-library_at_lists.haxx.se>
> Subject: [EXTERNAL] Re: CPU usage since 8.7.1
>
> On Tue, Oct 28, 2025 at 05:32:27PM +0100, Daniel Stenberg wrote:
> D> On Tue, 28 Oct 2025, Gleb Smirnoff via curl-library wrote:
> D>
> D> > So here I am raising this question. What needs to be done to get
> D> > CPU savings of 8.6.0 level and how can we help here?
> D>
> D> As recall things, the change that introduced the new (current)
> D> behavior was done to enhance the accuracy of the rate limit logic, at
> D> the expense of CPU.
> D>
> D> The change was pushed for by someone who argued that the increased
> D> precision was valuable for them if I remember correctly.
> D>
> D> Can we find a way to have both ways and let the user select which?
> D> Maybe for example by maybe something as silly and simple as doing
> D> CURLOPT_MAX_RECV_SPEED_LARGE % 10000 and if that is non-zero, higher
> D> precision is wanted.
>
> I added Stefan to explicit Cc. Since he very recently added 24badd29f titled "limit-rate revisited" (of course this conlicted with out patch again), he definitely should have an opinion on how this can be solved.
>
> --
> Gleb Smirnoff
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2025-10-30