Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

--write-out values for total_time, download_size, and speed_download do not agree #7017

Closed
jbromley opened this issue May 5, 2021 · 4 comments
Closed

--write-out values for total_time, download_size, and speed_download do not agree #7017

jbromley opened this issue May 5, 2021 · 4 comments
Assignees
@bagder

Comments

@jbromley
Copy link

jbromley commented May 5, 2021

I did this

Running on Ubuntu 20.10 using curl 7.76.1 (both from Ubuntu's repo and compiled from source) I was collecting some download speed statistics using the --write-out options with the following command

curl -s -w "%{url_effective},%{time_total},%{size_download},%{speed_download}\n" -o /dev/null -L ${f}

where ${f} is the file being downloaded.

The following shows a sample of the output I received:

http://localhost:8000/html/index.html,0.001196,4682,4682000.000
http://localhost:8000/downloads/index.html,0.001477,469,469000.000
http://localhost:8000/downloads/1k.bin,0.001758,1024,1024000.000
http://localhost:8000/downloads/2k.bin,0.001454,2048,2048000.000
http://localhost:8000/downloads/4k.bin,0.001581,4096,4096000.000
http://localhost:8000/downloads/8k.bin,0.001175,8192,8192000.000

Note that the value reported for speed_download is not equal to the size_download divided by the time_total. Instead it appears that the time_total has been truncated to the millisecond before calculating the speed_download. On my system, clock_getres for the monotonic clock indicates a 1 nanosecond resolution, so I have no reason to believe the printed times (to microsecond resolution) are incorrect.

I expected the following

I expect one of two things depending on whether or not the resolution of time measurements is 1 millisecond or 1 microsecond.

If the resolution for times is 1 millisecond, I would expect times to be printed with no more than three decimal places (i.e. millisecond resolution) and in this case the speed_download number is correct.

If the resolution for times is 1 microsecond, as it appears to be from the six places after the decimal, I expect speed_download to use the full value of the time for its calculation rather than the time truncated to milliseconds.

curl/libcurl version

Ubuntu version:
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh/0.9.3/openssl/zlib nghttp2/1.41.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

Compiled from source version:
curl 7.76.1-DEV (x86_64-pc-linux-gnu) libcurl/7.76.1-DEV OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.8.0 nghttp2/1.41.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM NTLM_WB PSL SSL TLS-SRP UnixSockets

operating system

Linux yemaya 5.11.0-7614-generic #15~1618626693~20.10~ecb25cd-Ubuntu SMP Thu Apr 22 16:00:45 UTC x86_64 x86_64 x86_64 GNU/Linux
@jbromley
Copy link
Author

jbromley commented May 5, 2021 •
edited

I did a little more digging and it appears the culprit might be the tvdiff function in tool_util.c. It converts everything to milliseconds when computing the difference of two timeval structs (which themselves have microsecond resolution). This function is used at tool_progress.c line 266 to calculate what seems to be the download speed.

I feel like I could fix this, but I am concerned that perhaps there are many places where it is assumed the time difference is in milliseconds.

@bagder
Copy link
Member

bagder commented May 6, 2021

I don't think it is at all related to tvdiff. The value for speed_download is extracted from libcurl using CURLINFO_SPEED_DOWNLOAD_T.

In libcurl, that value is calculated here:

curl/lib/progress.c

Lines 386 to 390 in d698c70

/* The average download speed this far */
if(dl < CURL_OFF_T_MAX/1000)
data->progress.dlspeed = (dl * 1000 / (timespent_ms>0?timespent_ms:1));
else
data->progress.dlspeed = (dl / (timespent>0?timespent:1));

It does indeed use millisecond precision there.

I'll create a PR with what I believe could be an appropriate fix would be neat if you could try it out!

bagder added a commit that referenced this issue May 6, 2021
@bagder
progress: when possible, calculate transfer speeds with microseconds
ca752e0 
... this improves precision, especially for transfers in the few or even
sub millisecond range.

Reported-by: J. Bromley
Fixes #7017
@bagder bagder mentioned this issue May 6, 2021
Closed
@bagder bagder self-assigned this May 6, 2021
@jbromley
Copy link
Author

jbromley commented May 6, 2021

I can definitely test out any fixes you might make.

@bagder
Copy link
Member

bagder commented May 6, 2021

Then check out #7020!

@bagder bagder closed this as completed in 6802762 May 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bagder bagder

Labels
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

progress: when possible, calculate transfer speeds with microseconds curl/curl
2 participants
@bagder @jbromley