curl / Mailing Lists / curl-library / Single Mail

Buy commercial curl support from WolfSSL. We help you work out your issues, debug your libcurl applications, use the API, port to new platforms, add new features and more. With a team lead by the curl founder himself.

Re: Large download speed deviations for CURLOPT_MAX_RECV_SPEED_LARGE in 8.4.0

Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]

From: Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>
Date: Fri, 1 Dec 2023 07:32:48 +0000

I tried the latest master branch with Daniel's fix, and I still see large deviations (up to 20%) from the speed limit set by CURLOPT_MAX_RECV_SPEED_LARGE.
The deviations depend on a network speed and they get bigger when the speed limit gets far from the network speed.

The 7.84.0 in my tests always had speed deviations from the speed limit < 1% regardless how far the speed limit was from the network speed,
which made speed throttling in 7.84 quite precise.

In my tests, I downloaded a 5 MB file from some local network server using different network speeds.
The test client was running on Windows (mentioning it, just in case).

Here is the snippet of the results:
----------------------------------------------- [master, ToT] ----------------------------------------------
Network speed: Link-local (~700 Mbps)
Url=https://192.168.0.6/Data/file_5M.bin, max_speed=8000000 bps (1000000 B/s)
time=6479 ms, dnld=5242880 B, speed=6473428 bps, spd_diff=-1526572 bps, pct=-19.1 %
time=6520 ms, dnld=5242880 B, speed=6432804 bps, spd_diff=-1567196 bps, pct=-19.6 %
time=6522 ms, dnld=5242880 B, speed=6430949 bps, spd_diff=-1569051 bps, pct=-19.6 %
time=5461 ms, dnld=5242880 B, speed=7679824 bps, spd_diff=-320176 bps, pct=-4.0 %
time=6502 ms, dnld=5242880 B, speed=6450361 bps, spd_diff=-1549639 bps, pct=-19.4 %...
...
Url=https://192.168.0.6/Data/file_5M.bin, max_speed=16000000 bps (2000000 B/s)
time=2681 ms, dnld=5242880 B, speed=15639709 bps, spd_diff=-360291 bps, pct=-2.3 %
time=2619 ms, dnld=5242880 B, speed=16012712 bps, spd_diff=12712 bps, pct=0.1 %
time=2672 ms, dnld=5242880 B, speed=15692677 bps, spd_diff=-307323 bps, pct=-1.9 %
time=2680 ms, dnld=5242880 B, speed=15646469 bps, spd_diff=-353531 bps, pct=-2.2 %
time=2420 ms, dnld=5242880 B, speed=17329529 bps, spd_diff=1329529 bps, pct=8.3 %
...

Network speed: [50 mbps]
Url=https://192.168.0.6/Data/file_5M.bin, max_speed=8000000 bps (1000000 B/s)
time=6672 ms, dnld=5242880 B, speed=6286245 bps, spd_diff=-1713755 bps, pct=-21.4 %
time=6697 ms, dnld=5242880 B, speed=6262587 bps, spd_diff=-1737413 bps, pct=-21.7 %
time=6701 ms, dnld=5242880 B, speed=6258471 bps, spd_diff=-1741529 bps, pct=-21.8 %
time=6114 ms, dnld=5242880 B, speed=6860144 bps, spd_diff=-1139856 bps, pct=-14.2 %
time=6591 ms, dnld=5242880 B, speed=6363237 bps, spd_diff=-1636763 bps, pct=-20.5 %
...
Url=https://192.168.0.6/Data/file_5M.bin, max_speed=16000000 bps (2000000 B/s)
time=2701 ms, dnld=5242880 B, speed=15526749 bps, spd_diff=-473251 bps, pct=-3.0 %
time=2695 ms, dnld=5242880 B, speed=15559149 bps, spd_diff=-440851 bps, pct=-2.8 %
time=2352 ms, dnld=5242880 B, speed=17831222 bps, spd_diff=1831222 bps, pct=11.4 %
time=2368 ms, dnld=5242880 B, speed=17711377 bps, spd_diff=1711377 bps, pct=10.7 %
time=2574 ms, dnld=5242880 B, speed=16292677 bps, spd_diff=292677 bps, pct=1.8 %
...

Network speed: [20 mbps]
Url=https://192.168.0.6/Data/file_5M.bin, max_speed=8000000 bps (1000000 B/s)
time=5794 ms, dnld=5242880 B, speed=7238636 bps, spd_diff=-761364 bps, pct=-9.5 %
time=5795 ms, dnld=5242880 B, speed=7236710 bps, spd_diff=-763290 bps, pct=-9.5 %
time=5785 ms, dnld=5242880 B, speed=7250173 bps, spd_diff=-749827 bps, pct=-9.4 %
time=5751 ms, dnld=5242880 B, speed=7292374 bps, spd_diff=-707626 bps, pct=-8.8 %
time=5397 ms, dnld=5242880 B, speed=7771316 bps, spd_diff=-228684 bps, pct=-2.9 %
...
Url=https://192.168.0.6/Data/file_5M.bin, max_speed=16000000 bps (2000000 B/s)
time=2576 ms, dnld=5242880 B, speed=16280357 bps, spd_diff=280357 bps, pct=1.8 %
time=2800 ms, dnld=5242880 B, speed=14975959 bps, spd_diff=-1024041 bps, pct=-6.4 %
time=2564 ms, dnld=5242880 B, speed=16355235 bps, spd_diff=355235 bps, pct=2.2 %
time=2625 ms, dnld=5242880 B, speed=15972948 bps, spd_diff=-27052 bps, pct=-0.2 %
time=2730 ms, dnld=5242880 B, speed=15363233 bps, spd_diff=-636767 bps, pct=-4.0 %
...

----------------------------------------------- [7.84.0] ----------------------------------------------
Network speed: Link-local (~700 Mbps)
Url=https://192.168.0.6/Data/file_5M.bin, max_speed=8000000 bps (1000000 B/s)
time=5263 ms, dnld=5242880 B, speed=7968429 bps, spd_diff=-31571 bps, pct=-0.4 %
time=5270 ms, dnld=5242880 B, speed=7957354 bps, spd_diff=-42646 bps, pct=-0.5 %
time=5264 ms, dnld=5242880 B, speed=7967887 bps, spd_diff=-32113 bps, pct=-0.4 %
time=5273 ms, dnld=5242880 B, speed=7953080 bps, spd_diff=-46920 bps, pct=-0.6 %
time=5266 ms, dnld=5242880 B, speed=7964399 bps, spd_diff=-35601 bps, pct=-0.4 %
...
Url=https://192.168.0.6/Data/file_5M.bin, max_speed=16000000 bps (2000000 B/s)
time=2642 ms, dnld=5242880 B, speed=15871783 bps, spd_diff=-128217 bps, pct=-0.8 %
time=2640 ms, dnld=5242880 B, speed=15884568 bps, spd_diff=-115432 bps, pct=-0.7 %
time=2642 ms, dnld=5242880 B, speed=15873106 bps, spd_diff=-126894 bps, pct=-0.8 %
time=2636 ms, dnld=5242880 B, speed=15910425 bps, spd_diff=-89575 bps, pct=-0.6 %
time=2640 ms, dnld=5242880 B, speed=15884987 bps, spd_diff=-115013 bps, pct=-0.7 %
...

Network speed: [50 mbps]
Url=https://192.168.0.6/Data/file_5M.bin, max_speed=8000000 bps (1000000 B/s)
time=5261 ms, dnld=5242880 B, speed=7971586 bps, spd_diff=-28414 bps, pct=-0.4 %
time=5261 ms, dnld=5242880 B, speed=7971029 bps, spd_diff=-28971 bps, pct=-0.4 %
time=5270 ms, dnld=5242880 B, speed=7958651 bps, spd_diff=-41349 bps, pct=-0.5 %
time=5274 ms, dnld=5242880 B, speed=7952013 bps, spd_diff=-47987 bps, pct=-0.6 %
time=5271 ms, dnld=5242880 B, speed=7957216 bps, spd_diff=-42784 bps, pct=-0.5 %
...
Url=https://192.168.0.6/Data/file_5M.bin, max_speed=16000000 bps
time=2631 ms, dnld=5242880 B, speed=15938125 bps, spd_diff=-61875 bps, pct=-0.4 %
time=2632 ms, dnld=5242880 B, speed=15932475 bps, spd_diff=-67525 bps, pct=-0.4 %
time=2633 ms, dnld=5242880 B, speed=15923828 bps, spd_diff=-76172 bps, pct=-0.5 %
time=2635 ms, dnld=5242880 B, speed=15916738 bps, spd_diff=-83262 bps, pct=-0.5 %
time=2638 ms, dnld=5242880 B, speed=15895312 bps, spd_diff=-104688 bps, pct=-0.7 %
...

Network speed: [20 mbps]
Url=https://192.168.0.6/Data/file_5M.bin, max_speed=8000000 bps (1000000 B/s)
time=5259 ms, dnld=5242880 B, speed=7974473 bps, spd_diff=-25527 bps, pct=-0.3 %
time=5265 ms, dnld=5242880 B, speed=7966040 bps, spd_diff=-33960 bps, pct=-0.4 %
time=5279 ms, dnld=5242880 B, speed=7944491 bps, spd_diff=-55509 bps, pct=-0.7 %
time=5266 ms, dnld=5242880 B, speed=7964010 bps, spd_diff=-35990 bps, pct=-0.4 %
time=5267 ms, dnld=5242880 B, speed=7962112 bps, spd_diff=-37888 bps, pct=-0.5 %
...

Sorry for too much text, but these data clearly indicate that something was regressed from the 7.84 for
the CURLOPT_MAX_RECV_SPEED_LARGE option, and the speed throttling is not that good anymore.

In my application, I have 5% tolerance for speed throttling, which unfortunately I can't get with the latest libcurl builds.

Thanks,
Dmitry Karpov

-----Original Message-----
From: Daniel Stenberg <daniel_at_haxx.se>
Sent: Thursday, November 30, 2023 1:42 AM
To: Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>
Cc: Dmitry Karpov <dkarpov_at_roku.com>
Subject: [EXTERNAL] Re: Large download speed deviations for CURLOPT_MAX_RECV_SPEED_LARGE in 8.4.0

On Wed, 29 Nov 2023, Dmitry Karpov via curl-library wrote:

> Is it something known or a not previously noticed regression in 8.4.0?

When I looked into this now, I noticed a regression in multi_getsock() which makes libcurl re-calculate the waiting periods during rate-limiting way too often. That's wasteful and in my tests it made the limited speed differ more from the desired value than what it should otherwise do.

This said: I could not see a 20% deviation in my tests. I saw a much smaller one, that seems to be mostly gone with my fix. So I can't say for sure if this really is the problem you see or if there is an additional one.

Anyway, my proposed first fix is here:

https://github.com/curl/curl/pull/12430

If you apply that patch, then doing debug outputs in the
Curl_pgrsLimitWaitTime() function is much easier and way less "floody". If you keep seeing this problem we can then double-check the calculations there to check that they seem reasonable. They do for me.

-- 
  / daniel.haxx.se
  | Commercial curl support up to 24x7 is available!
  | Private help, bug fixes, support, ports, new features
  | https://curl.se/support.html
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2023-12-01

Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]