Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Large download speed deviations for CURLOPT_MAX_RECV_SPEED_LARGE in 8.4.0
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>
Date: Sat, 2 Dec 2023 00:03:26 +0000
No, buffering is the same.
I explicitly set 1MB socket receive buffer to avoid any default buffering differences between 8.4. and 7.84.
The deviations are not sensitive to file sizes. They slightly vary between file sizes, but still are quite large for all file sizes.
For example:
Speed limit test [LAN (~700 mbps)]
**** 50 MB ****
Url=http://192.168.0.6/Data/file_50M.bin, max_speed=8000000 bps
time=63451 ms, dnld=52428800 B, speed=6610292 bps, spd_diff=-1389708 bps, pct=-17.4 %
time=63601 ms, dnld=52428800 B, speed=6594698 bps, spd_diff=-1405302 bps, pct=-17.6 %
time=64145 ms, dnld=52428800 B, speed=6538777 bps, spd_diff=-1461223 bps, pct=-18.3 %
Url=http://192.168.0.6/Data/file_50M.bin, max_speed=16000000 bps
time=34411 ms, dnld=52428800 B, speed=12188838 bps, spd_diff=-3811162 bps, pct=-23.8 %
time=34331 ms, dnld=52428800 B, speed=12217023 bps, spd_diff=-3782977 bps, pct=-23.6 %
**** 10 MB ****
Url=http://192.168.0.6/Data/file_10M.bin, max_speed=8000000 bps
time=12958 ms, dnld=10485760 B, speed=6473605 bps, spd_diff=-1526395 bps, pct=-19.1 %
time=12865 ms, dnld=10485760 B, speed=6520179 bps, spd_diff=-1479821 bps, pct=-18.5 %
Url=http://192.168.0.6/Data/file_10M.bin, max_speed=16000000 bps
time=6759 ms, dnld=10485760 B, speed=12409340 bps, spd_diff=-3590660 bps, pct=-22.4 %
time=6670 ms, dnld=10485760 B, speed=12576294 bps, spd_diff=-3423706 bps, pct=-21.4 %
…
Also, the size values in the “write function” are the same for both 8.4. and 7.84.
And total transfer times and speeds were measured by libcurl correctly and match what I measured myself using timers,
so it doesn’t look like a performance issue.
The factor which affects the deviations most is a difference between the network speed and the speed limit.
The less the difference the less the deviations.
This makes me think that probably current transfer speed evaluation (needed to calculate a transfer pause time to meet the speed limit)
somehow regressed in 8.4 (or earlier) and lost the precision it had in 7.84.
I can’t run tests on Linux at this moment or do a more advanced debugging as you suggested,
but I don’t think that the timing calculations really differ between Windows or Linux.
If I don’t use a speed limit, then I don’t see any performance differences between 8.4. and 7.84 for local network transfers,
so, I guess it is just timing calculations that need to be debugged.
I will try to spend more time debugging it next week, but at least on Windows this issue is quite visible with my test settings.
(local network (700 Mbps, 100 Mbps, 20 Mbps speeds), and speed limits (8 Mbps and 16 Mbps) ).
I tried multiple local network setups and different Windows clients and saw the same issue.
Thanks,
Dmitry Karpov
From: curl-library <curl-library-bounces_at_lists.haxx.se> On Behalf Of Timothe Litt via curl-library
Sent: Friday, December 1, 2023 1:56 AM
To: curl-library_at_lists.haxx.se
Cc: Timothe Litt <litt_at_acm.org>
Subject: [EXTERNAL] Re: Large download speed deviations for CURLOPT_MAX_RECV_SPEED_LARGE in 8.4.0
On 01-Dec-23 02:32, Dmitry Karpov via curl-library wrote:
In my tests, I downloaded a 5 MB file from some local network server using different network speeds.
The test client was running on Windows (mentioning it, just in case).
Random thought. Symptoms sound like different buffering. 5MB isn't all that big, what with all the buffering and today's network speeds. To get some clues: (a) is the deviation sensitive to file size? e.g., same deviation with 50MB? (b) do you see the same deviations from a Linux client? - if so, does strace show different setsockopt calls, perhaps setting buffer sizes? (c) Are you using the CURLOPT_WRITEFUNCTION? If so, is it getting the same 'nmemb' values? (if not, you can add one that does an fwrite() & log the nmemb values...) (d) If you enable VERBOSE and DEBUGFUNCTION and log to a (preallocated to minimize timing) memory buffer and print at exit, and the deviations don't go away: what's different in the log? (e) Does the actual elapsed time (measured by the OS/application) differ significantly? If so, it's not a timing measurement issue; rather it's a performance issue.
In all the above "different" means between libcurl versions.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
Date: Sat, 2 Dec 2023 00:03:26 +0000
No, buffering is the same.
I explicitly set 1MB socket receive buffer to avoid any default buffering differences between 8.4. and 7.84.
The deviations are not sensitive to file sizes. They slightly vary between file sizes, but still are quite large for all file sizes.
For example:
Speed limit test [LAN (~700 mbps)]
**** 50 MB ****
Url=http://192.168.0.6/Data/file_50M.bin, max_speed=8000000 bps
time=63451 ms, dnld=52428800 B, speed=6610292 bps, spd_diff=-1389708 bps, pct=-17.4 %
time=63601 ms, dnld=52428800 B, speed=6594698 bps, spd_diff=-1405302 bps, pct=-17.6 %
time=64145 ms, dnld=52428800 B, speed=6538777 bps, spd_diff=-1461223 bps, pct=-18.3 %
Url=http://192.168.0.6/Data/file_50M.bin, max_speed=16000000 bps
time=34411 ms, dnld=52428800 B, speed=12188838 bps, spd_diff=-3811162 bps, pct=-23.8 %
time=34331 ms, dnld=52428800 B, speed=12217023 bps, spd_diff=-3782977 bps, pct=-23.6 %
**** 10 MB ****
Url=http://192.168.0.6/Data/file_10M.bin, max_speed=8000000 bps
time=12958 ms, dnld=10485760 B, speed=6473605 bps, spd_diff=-1526395 bps, pct=-19.1 %
time=12865 ms, dnld=10485760 B, speed=6520179 bps, spd_diff=-1479821 bps, pct=-18.5 %
Url=http://192.168.0.6/Data/file_10M.bin, max_speed=16000000 bps
time=6759 ms, dnld=10485760 B, speed=12409340 bps, spd_diff=-3590660 bps, pct=-22.4 %
time=6670 ms, dnld=10485760 B, speed=12576294 bps, spd_diff=-3423706 bps, pct=-21.4 %
…
Also, the size values in the “write function” are the same for both 8.4. and 7.84.
And total transfer times and speeds were measured by libcurl correctly and match what I measured myself using timers,
so it doesn’t look like a performance issue.
The factor which affects the deviations most is a difference between the network speed and the speed limit.
The less the difference the less the deviations.
This makes me think that probably current transfer speed evaluation (needed to calculate a transfer pause time to meet the speed limit)
somehow regressed in 8.4 (or earlier) and lost the precision it had in 7.84.
I can’t run tests on Linux at this moment or do a more advanced debugging as you suggested,
but I don’t think that the timing calculations really differ between Windows or Linux.
If I don’t use a speed limit, then I don’t see any performance differences between 8.4. and 7.84 for local network transfers,
so, I guess it is just timing calculations that need to be debugged.
I will try to spend more time debugging it next week, but at least on Windows this issue is quite visible with my test settings.
(local network (700 Mbps, 100 Mbps, 20 Mbps speeds), and speed limits (8 Mbps and 16 Mbps) ).
I tried multiple local network setups and different Windows clients and saw the same issue.
Thanks,
Dmitry Karpov
From: curl-library <curl-library-bounces_at_lists.haxx.se> On Behalf Of Timothe Litt via curl-library
Sent: Friday, December 1, 2023 1:56 AM
To: curl-library_at_lists.haxx.se
Cc: Timothe Litt <litt_at_acm.org>
Subject: [EXTERNAL] Re: Large download speed deviations for CURLOPT_MAX_RECV_SPEED_LARGE in 8.4.0
On 01-Dec-23 02:32, Dmitry Karpov via curl-library wrote:
In my tests, I downloaded a 5 MB file from some local network server using different network speeds.
The test client was running on Windows (mentioning it, just in case).
Random thought. Symptoms sound like different buffering. 5MB isn't all that big, what with all the buffering and today's network speeds. To get some clues: (a) is the deviation sensitive to file size? e.g., same deviation with 50MB? (b) do you see the same deviations from a Linux client? - if so, does strace show different setsockopt calls, perhaps setting buffer sizes? (c) Are you using the CURLOPT_WRITEFUNCTION? If so, is it getting the same 'nmemb' values? (if not, you can add one that does an fwrite() & log the nmemb values...) (d) If you enable VERBOSE and DEBUGFUNCTION and log to a (preallocated to minimize timing) memory buffer and print at exit, and the deviations don't go away: what's different in the log? (e) Does the actual elapsed time (measured by the OS/application) differ significantly? If so, it's not a timing measurement issue; rather it's a performance issue.
In all the above "different" means between libcurl versions.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2023-12-02