Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The warning output gets mangled by the progress meter #4422

Closed
combor opened this issue Sep 25, 2019 · 17 comments
Closed

The warning output gets mangled by the progress meter #4422

combor opened this issue Sep 25, 2019 · 17 comments
Labels
cmdline tool KNOWN_BUGS material

Comments

@combor
Copy link

combor commented Sep 25, 2019 •
edited

I did this

Hi, I'm having problems hiding the progress bar but keeping the warings on the screen.
With such command I'm getting only the last error:

curl -Lkf --connect-timeout 5 \
    --max-time 10 \
    --retry 5 \
    --retry-max-time 40 \
    --show-error \
    --silent \
    --output /tmp/response \
    -w "Requested URL: %{url_effective}\nResponse code: %{http_code}\n" \
    'https://httpstat.us/503'

Requested URL: https://httpstat.us/503
Response code: 503
curl: (22) The requested URL returned error: 503 Service Unavailable

If I remove --silent I'm getting this:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
Warning: Transient problem: HTTP error Will retry in 1 seconds. 5 retries
Warning: left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
Warning: Transient problem: HTTP error Will retry in 2 seconds. 4 retries
Warning: left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
Warning: Transient problem: HTTP error Will retry in 4 seconds. 3 retries
Warning: left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
Warning: Transient problem: HTTP error Will retry in 8 seconds. 2 retries
Warning: left.
^C

Or this ( no new lines before Warning) - depends on the error type:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Transient problem: timeout Will retry in 1 seconds. 5 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Transient problem: timeout Will retry in 2 seconds. 4 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Transient problem: timeout Will retry in 4 seconds. 3 retries left.
^C

How can I hide the progress bar? After some googling it seems like the most wanted feature on the internet :)

Maybe the new flag can be introduced? Something like --do-not-ever-show-me-the-progress-bar-as-i-know-what-im-doing ?

I expected the following

Warning: Transient problem: timeout Will retry in 1 seconds. 5 retries left.
Warning: Transient problem: timeout Will retry in 2 seconds. 4 retries left.
Warning: Transient problem: timeout Will retry in 4 seconds. 3 retries left.
...
curl: (22) The requested URL returned error: 503 Service Unavailable

curl/libcurl version

curl 7.54.0 (x86_64-apple-darwin18.0) libcurl/7.54.0 LibreSSL/2.6.5 zlib/1.2.11 nghttp2/1.24.1
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy

operating system

Darwin GBML242731 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64
@bagder
Copy link
Member

bagder commented Sep 25, 2019

Hi!

Thanks for your interest in curl and related manners, but you seem to have asked a question or help, and this is an issue tracker that we reserve for dealing with issues/bugs really.

Please ask questions and ask for support etc on the dedicated mailing lists. You will also reach a lot more people that way!

You clearly noticed that -s / --silent removed the progress bar so for being "like the most wanted feature on the internet :)" it was rather simple, right?

Now you want to keep some "talk" output and remove other and that's a different matter and we don't really let the user pick-and-choose among all of them. We have normal mode, silent mode and verbose mode. You need to pick one of them.

Or work on adding some sort of new selection option that isn't currently available...

@bagder bagder closed this as completed Sep 25, 2019
@combor
Copy link
Author

combor commented Sep 25, 2019

Thanks for the quick response. As you probably noticed this is a bug report as well. Please check the second output from the non-silenced curl. The output with warings doesn't have new lines which makes it non-readable. Is this something that you consider as a bug?

@bagder bagder reopened this Sep 25, 2019
@bagder
Copy link
Member

bagder commented Sep 25, 2019

I guess it is a bug yes.

The problem is of course that we have both the progress meter and the warnings getting shown and it's a bad mix since the progress meter will destroy anything else that is also shown... It will never be "pretty".

@bagder bagder changed the title How to hide progress stats? The warning output gets mangled by the progress meter Sep 25, 2019
@combor
Copy link
Author

combor commented Sep 25, 2019

Thanks, sounds like a good solution would be to hide the progress meter.

@bagder
Copy link
Member

bagder commented Sep 25, 2019

When?

@combor
Copy link
Author

combor commented Sep 25, 2019

Maybe when a flag is specified to hide it ? I mean, if it's hard to make it pretty by default I recon It would be desirable to let user to choose what to output.

bagder added a commit that referenced this issue Oct 7, 2019
@bagder
curl: --no-progress-meter
981c3bb 
New option that allows a user to ONLY switch off curl's progress meter
and leave everything else in "talkative" mode.

Fixes #4422
@bagder bagder mentioned this issue Oct 7, 2019
Closed
@bagder
Copy link
Member

bagder commented Oct 8, 2019 •
edited

@combor, I presume then you'd be happy with something in the style of #4470 which allows users to switch off the progress-bar only, but not making curl silent otherwise?

@combor
Copy link
Author

combor commented Oct 8, 2019

Yes, that would be perfect. Thanks @bagder

@jay
Copy link
Member

jay commented Oct 8, 2019

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Transient problem: timeout Will retry in 4 seconds. 3 retries left.
^C

I can't reproduce the missing newline using master in Windows 7. Do you see that when you use master?

@bagder bagder closed this as completed in 93373a9 Oct 8, 2019
@combor
Copy link
Author

combor commented Oct 8, 2019

@jay try curling a host that will refuse connections. I can reproduce it while trying to connect to http://localhost:1234

 curl -Lkf --connect-timeout 5     --max-time 10     --retry 5     --retry-max-time 40     --show-error       --output /tmp/response     -w "Requested URL: %{url_effective}\nResponse code: %{http_code}\n"     'https://localhost:1234'

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Requested URL: https://localhost:1234/
Response code: 000
curl: (7) Failed to connect to localhost port 1234: Connection refused

I havent tried master though.

@jay
Copy link
Member

jay commented Oct 8, 2019

Interesting. I can't reproduce that here. It doesn't seem to affect Windows or maybe it's a timing issue.

@combor
Copy link
Author

combor commented Oct 8, 2019

I just tried on windows and it actually doesn't do that while connection is refused. But by mistake I have copied exactly same command from macOS and because windows doesn't like single quotes it can't resolve localhost. Now it has no new lines in the output. Try executing exactly this:

c:\>curl.exe -Lkf --connect-timeout 5     --max-time 10     --retry 5     --retry-max-time 40     --show-error       --output /tmp/response     -w "Requested URL: %{url_effective}\nResponse code: %{http_code}\n"     'https://localhost:1234'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: 'https
Warning: Transient problem: timeout Will retry in 1 seconds. 5 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: 'https
Warning: Transient problem: timeout Will retry in 2 seconds. 4 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: 'https
Warning: Transient problem: timeout Will retry in 4 seconds. 3 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: 'https
Warning: Transient problem: timeout Will retry in 8 seconds. 2 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: 'https
Warning: Transient problem: timeout Will retry in 16 seconds. 1 retries left.

This is latest curl v 7.66.0

The other interesting thing is when connection is refused the expotential backoff seems to not be triggered. Is this intentional?

@jay
Copy link
Member

jay commented Oct 8, 2019

Ok I see. That doesn't work (look at your error message) because of the single quotes and you must have --retry-connrefused in your curlrc. Here's a minimal repro that shows me the issue in older curl versions but not in newer ones.

curl 7.59:

> curl -V && curl --retry-connrefused --retry 3 --output NUL https://localhost:1234
curl 7.59.0-DEV (i686-w64-mingw32) libcurl/7.59.0-DEV mbedTLS/2.7.0 zlib/1.2.11 libssh2/1.8.0 nghttp2/1.29.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS Debug TrackMemory IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz HTTP2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Transient problem: connection refused Will retry in 1 seconds. 3
Warning: retries left.
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0Warning: Transient problem: connection refused Will retry in 2 seconds. 2
Warning: retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Transient problem: connection refused Will retry in 4 seconds. 1
Warning: retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 1234: Connection refused

master:

> curld -V && curld --retry-connrefused --retry 3 --output NUL https://localhost:1234
curl 7.67.0-DEV (i386-pc-win32) libcurl/7.67.0-DEV OpenSSL/1.0.2s nghttp2/1.39.1
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS Debug HTTP2 HTTPS-proxy IPv6 Largefile NTLM SSL
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
curl: (7) Failed to connect to localhost port 1234: Connection refused
Warning: Transient problem: connection refused Will retry in 1 seconds. 3
Warning: retries left.
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
curl: (7) Failed to connect to localhost port 1234: Connection refused
Warning: Transient problem: connection refused Will retry in 2 seconds. 2
Warning: retries left.
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
curl: (7) Failed to connect to localhost port 1234: Connection refused
Warning: Transient problem: connection refused Will retry in 4 seconds. 1
Warning: retries left.
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
curl: (7) Failed to connect to localhost port 1234: Connection refused

@combor
Copy link
Author

combor commented Oct 8, 2019

Ok, so it seems that new lines problem exists only when using non existent domain. Here an example:

curl.exe -V && curl.exe --retry-connrefused --retry 3 --output NUL https://non.existent.domain
curl 7.66.0 (x86_64-pc-win32) libcurl/7.66.0 OpenSSL/1.1.1d (Schannel) zlib/1.2.11 brotli/1.0.7 WinIDN libssh2/1.9.0 nghttp2/1.39.2
Release-Date: 2019-09-11
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP brotli libz

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: non.existent.domain
Warning: Transient problem: timeout Will retry in 1 seconds. 3 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: non.existent.domain
Warning: Transient problem: timeout Will retry in 2 seconds. 2 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: non.existent.domain
Warning: Transient problem: timeout Will retry in 4 seconds. 1 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: non.existent.domain

@jay
Copy link
Member

jay commented Oct 8, 2019 •
edited

Ok. I see Curl_pgrsDone (which adds the newline when the progress meter is done) is not called in some places. If the host cannot be resolved then it is not called.

Curl_pgrsDone is called by multi_done which is the cleanup but multi_done is not called for early states in the state machine. If I add it to CURLM_STATE_WAITRESOLVE state it will solve at least something. I don't know if it is safe for all states though, @bagder.

diff --git a/lib/multi.c b/lib/multi.c
index 6dfe884..4b8fd78 100755
--- a/lib/multi.c
+++ b/lib/multi.c
@@ -1616,6 +1616,7 @@ static CURLMcode multi_runsingle(struct Curl_multi *multi,
       if(result) {
         /* failure detected */
         stream_error = TRUE;
+        multi_done(data, result, TRUE);
         break;
       }
     }
> curld -o NUL --retry 1 DOESNOTEXIST
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
curl: (6) Could not resolve host: DOESNOTEXIST
Warning: Transient problem: timeout Will retry in 1 seconds. 1 retries left.
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
curl: (6) Could not resolve host: DOESNOTEXIST

@jay jay reopened this Oct 8, 2019
@bagder
Copy link
Member

bagder commented Nov 8, 2019

multi_done() is meant to be the counter-part to multi_do(). If it stops before do is called, I think it makes sense that done isn't called... but maybe we need to change that.

@bagder
Copy link
Member

bagder commented Feb 14, 2020

I consider this bug very minor and won't work on it myself. It's been open 3+ months without activity. I think we should close it and I'm not sure it is even worth documenting in KNOWN_BUGS...

@bagder bagder closed this as completed Mar 17, 2020
@lock lock bot locked as resolved and limited conversation to collaborators Jun 24, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
cmdline tool KNOWN_BUGS material
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

curl: --no-progress-meter curl/curl
3 participants
@combor @bagder @jay