Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
Increase in CPU usage in 8.7.1 vs 8.6.0 for rate-limited downloads
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: David Pfitzner via curl-library <curl-library_at_lists.haxx.se>
Date: Wed, 15 May 2024 16:26:18 +0930
Hi,
I've noticed that from curl 8.6.0 to 8.7.1, the CPU usage when doing
rate-limited downloads with libcurl has increased significantly. This is
most noticeable when making multiple (rate-limited) downloads in parallel,
for large files. (I've seen this in a custom application using libcurl, and
then reproduced using the 'curl' tool.)
For example, running 'curl' with the following (where the file a32 contains
32 copies of " -o /dev/null http://127.0.0.1:9999/speed10", a test resource
with size=46450490):
#curl-8.6.0:
for x in 1 2 3 4 5 ; do /usr/bin/time ../curl-8_6_0 --parallel
--parallel-immediate --no-progress-meter --limit-rate 10485760 `cat ./a32`
; done
5.02 real 0.20 user 0.67 sys
4.95 real 0.15 user 0.68 sys
5.02 real 0.25 user 0.63 sys
5.00 real 0.21 user 0.63 sys
4.49 real 0.23 user 0.62 sys
# curl-8.7.1:
for x in 1 2 3 4 5 ; do /usr/bin/time ../curl-8_7_1 --parallel
--parallel-immediate --no-progress-meter --limit-rate 10485760 `cat ./a32`
; done
4.51 real 0.52 user 0.65 sys
5.01 real 0.55 user 0.63 sys
5.02 real 0.56 user 0.59 sys
4.96 real 0.42 user 0.59 sys
5.01 real 0.37 user 0.71 sys
Notice the increase in user-space CPU usage (almost doubled). (Of course
the actual effect can vary depending on many factors - the number of
parallel downloads, the specified rate limit, file sizes, network speed,
etc.)
From testing, this change appears to be due to this commit:
https://github.com/curl/curl/commit/db5c9f4f9e0779b49624752b135281a0717b277b
That change was to make rate limiting more accurate, and perhaps the
increase in CPU usage is a necessary (and expected?) cost of doing so. But,
in my use case the rate limiting accuracy in 8.6.0 was "good enough", and I
would prefer not to incur the increase in CPU usage seen in 8.7.1. But, I'm
not sure how to do that. Perhaps it would be useful for a user of libcurl
to be able to (somehow) control this tradeoff between rate-limiting
accuracy and CPU usage?
Regards,
David Pfitzner
Date: Wed, 15 May 2024 16:26:18 +0930
Hi,
I've noticed that from curl 8.6.0 to 8.7.1, the CPU usage when doing
rate-limited downloads with libcurl has increased significantly. This is
most noticeable when making multiple (rate-limited) downloads in parallel,
for large files. (I've seen this in a custom application using libcurl, and
then reproduced using the 'curl' tool.)
For example, running 'curl' with the following (where the file a32 contains
32 copies of " -o /dev/null http://127.0.0.1:9999/speed10", a test resource
with size=46450490):
#curl-8.6.0:
for x in 1 2 3 4 5 ; do /usr/bin/time ../curl-8_6_0 --parallel
--parallel-immediate --no-progress-meter --limit-rate 10485760 `cat ./a32`
; done
5.02 real 0.20 user 0.67 sys
4.95 real 0.15 user 0.68 sys
5.02 real 0.25 user 0.63 sys
5.00 real 0.21 user 0.63 sys
4.49 real 0.23 user 0.62 sys
# curl-8.7.1:
for x in 1 2 3 4 5 ; do /usr/bin/time ../curl-8_7_1 --parallel
--parallel-immediate --no-progress-meter --limit-rate 10485760 `cat ./a32`
; done
4.51 real 0.52 user 0.65 sys
5.01 real 0.55 user 0.63 sys
5.02 real 0.56 user 0.59 sys
4.96 real 0.42 user 0.59 sys
5.01 real 0.37 user 0.71 sys
Notice the increase in user-space CPU usage (almost doubled). (Of course
the actual effect can vary depending on many factors - the number of
parallel downloads, the specified rate limit, file sizes, network speed,
etc.)
From testing, this change appears to be due to this commit:
https://github.com/curl/curl/commit/db5c9f4f9e0779b49624752b135281a0717b277b
That change was to make rate limiting more accurate, and perhaps the
increase in CPU usage is a necessary (and expected?) cost of doing so. But,
in my use case the rate limiting accuracy in 8.6.0 was "good enough", and I
would prefer not to incur the increase in CPU usage seen in 8.7.1. But, I'm
not sure how to do that. Perhaps it would be useful for a user of libcurl
to be able to (somehow) control this tradeoff between rate-limiting
accuracy and CPU usage?
Regards,
David Pfitzner
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2024-05-15