Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
Re: CPU usage since 8.7.1
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>
Date: Thu, 30 Oct 2025 19:03:51 +0000
> We'd have to revisit rate-limited/paused transfers in the future. Your measurements show that this could be better.
> Not sure when we'll find the time for that. So, I cannot promise anything.
Yes, it would be great if it could be addressed in the future as the inaccuracy with a tendency to give more bandwidth to the rate limited transfers
than the rate limit could be a significant regression for certain applications, where precise bandwidth distribution between transfers with different priorities
might be more valuable than a bit higher CPU utilization.
And the problem with the current implementation is that it produces too large deviations always using more bandwidth.
Maybe, it can be easily adjusted to be more biased toward using less bandwidth than the rate limit while still being not very accurate?
Thanks!
Dmitry Karpov
-----Original Message-----
From: Stefan Eissing <stefan_at_eissing.org>
Sent: Thursday, October 30, 2025 2:56 AM
To: libcurl development <curl-library_at_lists.haxx.se>
Cc: Dmitry Karpov <dkarpov_at_roku.com>
Subject: [EXTERNAL] Re: CPU usage since 8.7.1
[You don't often get email from stefan_at_eissing.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> Am 29.10.2025 um 22:12 schrieb Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>:
>
> I ran the same throttling tests on 8.16.0 that helped to find the speed throttling issue on 8.7, and I can tell that the throttling accuracy in 8.16.0 is significantly worse than in 8.6.0 and 8.7.1.
>
> My tests use 1MB downloads on network speeds: 700mbps, 50mbps and 20mbps, with 8mbps and 16mbps throttling speeds.
> The test measures the performed transfers speeds on 10 downloads and calculates average and max deviations from the rate limits.
>
> Here is the brief description of test results:
>
> 8.6.0 (and 8.7.1)
> Network speed: 700mbps, Rate limit: 8mbps, Avg deviation: -0.86%, Max deviation: -1.5%.
> Network speed: 700mbps, Rate limit: 16mbps, Avg deviation: -1.72%, Max deviation: -2.8%.
>
> Network speed: 50mbps, Rate limit: 8mbps, Avg deviation: -0.37%, Max deviation: -1%.
> Network speed: 50mbps, Rate limit: 16mbps, Avg deviation: - 1.64%, Max deviation: -3.2%.
>
> Network speed: 20mbps, Rate limit: 8mbps, Avg deviation: -0.6%, Max deviation: -1.1%.
> Network speed: 20mbps, Rate limit: 16mbps, Avg deviation: -1.37%, Max deviation: -1.7%.
>
> 8.16.0
> Network speed: 700mbps, Rate limit: 8mbps, Avg deviation: 4.65%, Max deviation: 13.7%.
> Network speed: 700mbps, Rate limit: 16mbps, Avg deviation: 6.69%, Max deviation: 17.7%.
>
> Network speed: 50mbps, Rate limit: 8mbps, Avg deviation: 7.03%, Max deviation: 10.2%.
> Network speed: 50mbps, Rate limit: 16mbps, Avg deviation: 9.99%, Max deviation: 15%.
>
> Network speed: 20mbps, Rate limit: 8mbps, Avg deviation: 10.89%, Max deviation: 15.5%.
> Network speed: 20mbps, Rate limit: 16mbps, Avg deviation: 5.34%, Max deviation: 14.5%.
>
> As we can see, the 8.6.0 (and 8.7.1) has almost perfect rate limit
> accuracy with a tendency to curb the download speed a little but below the limit, which is a good thing that makes the download speed unlikely to go beyond the limit.
>
> On the other hand, the 8.16.0 has wider deviations from the rate limits with a tendency to use more network bandwidth than the limit specifies.
> And the max deviations above 10% make it difficult to have reliable predictions of the throttled download speeds.
>
> If the almost perfect accuracy of 8.6.0/8.7.1 comes with a price of
> much higher CPU utilization, which maybe not acceptable for everyone,
> then maybe we need to adjust the rate limiting mechanism and make it to provide just 5% accuracy with a tendency to use less network bandwidth than the rate limit, when the precise accuracy is not possible.
>
> The tendency to use more bandwidth in the current mechanism creates
> problems when rate limits are specifically set to prevent low priority downloads from stealing network bandwidth from downloads with higher priority.
We'd have to revisit rate-limited/paused transfers in the future. Your measurements show that this could be better.
Not sure when we'll find the time for that. So, I cannot promise anything.
Cheers,
Stefan
>
> Thanks,
> Dmitry Karpov
>
>
>
>
> -----Original Message-----
> From: curl-library <curl-library-bounces_at_lists.haxx.se> On Behalf Of
> Dmitry Karpov via curl-library
> Sent: Tuesday, October 28, 2025 11:38 AM
> To: libcurl development <curl-library_at_lists.haxx.se>
> Cc: Dmitry Karpov <dkarpov_at_roku.com>
> Subject: [EXTERNAL] Re: CPU usage since 8.7.1
>
> It was me who reported it back then.
> The precision loss of the rated downloads was more than 10% in some cases, which was above our 5% tolerance.
>
> It caused problems in multi-download prioritized scenarios where rated downloads with less priority were stealing bandwidth from the downloads with higher priority.
>
> I am also concerned about CPU usage, but 10% rate precision deviation is too high.
> And in our case (video streaming application), the problem was that the prioritized video got worse quality and other problems which affected our users.
>
> Ideally, the optimal solution would be to avoid busy loops and using too many CPU cycles, and provide 5% rate accuracy.
>
> Thanks!
> Dmitry Karpov
>
> -----Original Message-----
> From: curl-library <curl-library-bounces_at_lists.haxx.se> On Behalf Of
> Gleb Smirnoff via curl-library
> Sent: Tuesday, October 28, 2025 11:11 AM
> To: Daniel Stenberg <daniel_at_haxx.se>; Stefan Eissing
> <stefan_at_eissing.org>
> Cc: Gleb Smirnoff <glebius_at_glebi.us>; Gleb Smirnoff via curl-library
> <curl-library_at_lists.haxx.se>
> Subject: [EXTERNAL] Re: CPU usage since 8.7.1
>
> On Tue, Oct 28, 2025 at 05:32:27PM +0100, Daniel Stenberg wrote:
> D> On Tue, 28 Oct 2025, Gleb Smirnoff via curl-library wrote:
> D>
> D> > So here I am raising this question. What needs to be done to get
> D> > CPU savings of 8.6.0 level and how can we help here?
> D>
> D> As recall things, the change that introduced the new (current)
> D> behavior was done to enhance the accuracy of the rate limit logic,
> D> at the expense of CPU.
> D>
> D> The change was pushed for by someone who argued that the increased
> D> precision was valuable for them if I remember correctly.
> D>
> D> Can we find a way to have both ways and let the user select which?
> D> Maybe for example by maybe something as silly and simple as doing
> D> CURLOPT_MAX_RECV_SPEED_LARGE % 10000 and if that is non-zero,
> D> higher precision is wanted.
>
> I added Stefan to explicit Cc. Since he very recently added 24badd29f titled "limit-rate revisited" (of course this conlicted with out patch again), he definitely should have an opinion on how this can be solved.
>
> --
> Gleb Smirnoff
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
Date: Thu, 30 Oct 2025 19:03:51 +0000
> We'd have to revisit rate-limited/paused transfers in the future. Your measurements show that this could be better.
> Not sure when we'll find the time for that. So, I cannot promise anything.
Yes, it would be great if it could be addressed in the future as the inaccuracy with a tendency to give more bandwidth to the rate limited transfers
than the rate limit could be a significant regression for certain applications, where precise bandwidth distribution between transfers with different priorities
might be more valuable than a bit higher CPU utilization.
And the problem with the current implementation is that it produces too large deviations always using more bandwidth.
Maybe, it can be easily adjusted to be more biased toward using less bandwidth than the rate limit while still being not very accurate?
Thanks!
Dmitry Karpov
-----Original Message-----
From: Stefan Eissing <stefan_at_eissing.org>
Sent: Thursday, October 30, 2025 2:56 AM
To: libcurl development <curl-library_at_lists.haxx.se>
Cc: Dmitry Karpov <dkarpov_at_roku.com>
Subject: [EXTERNAL] Re: CPU usage since 8.7.1
[You don't often get email from stefan_at_eissing.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> Am 29.10.2025 um 22:12 schrieb Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>:
>
> I ran the same throttling tests on 8.16.0 that helped to find the speed throttling issue on 8.7, and I can tell that the throttling accuracy in 8.16.0 is significantly worse than in 8.6.0 and 8.7.1.
>
> My tests use 1MB downloads on network speeds: 700mbps, 50mbps and 20mbps, with 8mbps and 16mbps throttling speeds.
> The test measures the performed transfers speeds on 10 downloads and calculates average and max deviations from the rate limits.
>
> Here is the brief description of test results:
>
> 8.6.0 (and 8.7.1)
> Network speed: 700mbps, Rate limit: 8mbps, Avg deviation: -0.86%, Max deviation: -1.5%.
> Network speed: 700mbps, Rate limit: 16mbps, Avg deviation: -1.72%, Max deviation: -2.8%.
>
> Network speed: 50mbps, Rate limit: 8mbps, Avg deviation: -0.37%, Max deviation: -1%.
> Network speed: 50mbps, Rate limit: 16mbps, Avg deviation: - 1.64%, Max deviation: -3.2%.
>
> Network speed: 20mbps, Rate limit: 8mbps, Avg deviation: -0.6%, Max deviation: -1.1%.
> Network speed: 20mbps, Rate limit: 16mbps, Avg deviation: -1.37%, Max deviation: -1.7%.
>
> 8.16.0
> Network speed: 700mbps, Rate limit: 8mbps, Avg deviation: 4.65%, Max deviation: 13.7%.
> Network speed: 700mbps, Rate limit: 16mbps, Avg deviation: 6.69%, Max deviation: 17.7%.
>
> Network speed: 50mbps, Rate limit: 8mbps, Avg deviation: 7.03%, Max deviation: 10.2%.
> Network speed: 50mbps, Rate limit: 16mbps, Avg deviation: 9.99%, Max deviation: 15%.
>
> Network speed: 20mbps, Rate limit: 8mbps, Avg deviation: 10.89%, Max deviation: 15.5%.
> Network speed: 20mbps, Rate limit: 16mbps, Avg deviation: 5.34%, Max deviation: 14.5%.
>
> As we can see, the 8.6.0 (and 8.7.1) has almost perfect rate limit
> accuracy with a tendency to curb the download speed a little but below the limit, which is a good thing that makes the download speed unlikely to go beyond the limit.
>
> On the other hand, the 8.16.0 has wider deviations from the rate limits with a tendency to use more network bandwidth than the limit specifies.
> And the max deviations above 10% make it difficult to have reliable predictions of the throttled download speeds.
>
> If the almost perfect accuracy of 8.6.0/8.7.1 comes with a price of
> much higher CPU utilization, which maybe not acceptable for everyone,
> then maybe we need to adjust the rate limiting mechanism and make it to provide just 5% accuracy with a tendency to use less network bandwidth than the rate limit, when the precise accuracy is not possible.
>
> The tendency to use more bandwidth in the current mechanism creates
> problems when rate limits are specifically set to prevent low priority downloads from stealing network bandwidth from downloads with higher priority.
We'd have to revisit rate-limited/paused transfers in the future. Your measurements show that this could be better.
Not sure when we'll find the time for that. So, I cannot promise anything.
Cheers,
Stefan
>
> Thanks,
> Dmitry Karpov
>
>
>
>
> -----Original Message-----
> From: curl-library <curl-library-bounces_at_lists.haxx.se> On Behalf Of
> Dmitry Karpov via curl-library
> Sent: Tuesday, October 28, 2025 11:38 AM
> To: libcurl development <curl-library_at_lists.haxx.se>
> Cc: Dmitry Karpov <dkarpov_at_roku.com>
> Subject: [EXTERNAL] Re: CPU usage since 8.7.1
>
> It was me who reported it back then.
> The precision loss of the rated downloads was more than 10% in some cases, which was above our 5% tolerance.
>
> It caused problems in multi-download prioritized scenarios where rated downloads with less priority were stealing bandwidth from the downloads with higher priority.
>
> I am also concerned about CPU usage, but 10% rate precision deviation is too high.
> And in our case (video streaming application), the problem was that the prioritized video got worse quality and other problems which affected our users.
>
> Ideally, the optimal solution would be to avoid busy loops and using too many CPU cycles, and provide 5% rate accuracy.
>
> Thanks!
> Dmitry Karpov
>
> -----Original Message-----
> From: curl-library <curl-library-bounces_at_lists.haxx.se> On Behalf Of
> Gleb Smirnoff via curl-library
> Sent: Tuesday, October 28, 2025 11:11 AM
> To: Daniel Stenberg <daniel_at_haxx.se>; Stefan Eissing
> <stefan_at_eissing.org>
> Cc: Gleb Smirnoff <glebius_at_glebi.us>; Gleb Smirnoff via curl-library
> <curl-library_at_lists.haxx.se>
> Subject: [EXTERNAL] Re: CPU usage since 8.7.1
>
> On Tue, Oct 28, 2025 at 05:32:27PM +0100, Daniel Stenberg wrote:
> D> On Tue, 28 Oct 2025, Gleb Smirnoff via curl-library wrote:
> D>
> D> > So here I am raising this question. What needs to be done to get
> D> > CPU savings of 8.6.0 level and how can we help here?
> D>
> D> As recall things, the change that introduced the new (current)
> D> behavior was done to enhance the accuracy of the rate limit logic,
> D> at the expense of CPU.
> D>
> D> The change was pushed for by someone who argued that the increased
> D> precision was valuable for them if I remember correctly.
> D>
> D> Can we find a way to have both ways and let the user select which?
> D> Maybe for example by maybe something as silly and simple as doing
> D> CURLOPT_MAX_RECV_SPEED_LARGE % 10000 and if that is non-zero,
> D> higher precision is wanted.
>
> I added Stefan to explicit Cc. Since he very recently added 24badd29f titled "limit-rate revisited" (of course this conlicted with out patch again), he definitely should have an opinion on how this can be solved.
>
> --
> Gleb Smirnoff
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2025-10-30