Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Issues with output
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: TD via curl-users <curl-users_at_lists.haxx.se>
Date: Mon, 12 Jun 2023 23:08:28 +0000 (UTC)
Thanks Daniel.Yes, 8.1.0 is the version.
The config file has an output file specified as part of the config file, so I thought that all the output for the site would be sent to the file, and the output from -w with the stats would then go to stdout, which I redirect in append mode. This works on the command line with a single curl command and the config file set in the command line. It also works with multiple curl commands, if I use the -o option on the command line set with a variable (-o $site.out), and then redirect the stdout to a second file for the stats.
The issue is that curl is not sending the output from curl itself to the output destination in the config file. Instead it sends both the output from -w and the output from curl to the redirected stdout intended for the -w data only. Is there some way to set curl so that it sends -w output to one file in append mode, and uses the file specified in the config file for the output of the curl command?
I know it is reading the URL from the config file, because that is the source of the URLs. I know some of the URLs are written to files specified in the config file.
Could it be that the behavior of curl if it fails to get the filename/file path from the config file it defaults back to stdout? If that's the case, then could there be some issue with the filename parsing (maybe my filenames are too long or being ignored? I have long file paths and long file names in the config file) Is there a maximum length for a filename or maximum number of directories deep for the path?
I really appreciate the help as this has me stumped! All the hard work everyone has done in the 8.x branch has made this perform substantially better than the previous versions...if I can just get the kinks worked out in my process :)
On Monday, June 12, 2023 at 05:12:05 PM EDT, Daniel Stenberg <daniel_at_haxx.se> wrote:
On Mon, 12 Jun 2023, TD via curl-users wrote:
> Using Curl 8.10 compiled with openSSL on Ubuntu 20.04.
You probably mean 8.1.0
> I know curl has threading options
You probably mean the parallel option. curl does many simultaneous transfers
in the same thread.
> If I use GNU curl, the output from the -w gets added to the output from the
> curl command and it all ends up in one file.
Because you send them both to stdout?
> cat $ListOfConfigFilesToPass |parallel -P3000 curl -s -w "$listOfOptions"
> --doh-url $DOH_URL -A $AGENT_STRING --connect-timeout 15 --max-time 60 -k -L
> --config {} >>$outputFileForStats 2>/dev/null It seems like there should be
> 3 distinct outputs from this:---stdout (the output of -w) ---stderr (blank
> because the -s option should suppress it, and anything else is being sent to
> /dev/null)---the output file specified in the config file.
>
> Am I interpreting this correctly?
Not exactly. The -w output goes to stdout and since you don't use --output,
the normal data is also output to stdout.
Date: Mon, 12 Jun 2023 23:08:28 +0000 (UTC)
Thanks Daniel.Yes, 8.1.0 is the version.
The config file has an output file specified as part of the config file, so I thought that all the output for the site would be sent to the file, and the output from -w with the stats would then go to stdout, which I redirect in append mode. This works on the command line with a single curl command and the config file set in the command line. It also works with multiple curl commands, if I use the -o option on the command line set with a variable (-o $site.out), and then redirect the stdout to a second file for the stats.
The issue is that curl is not sending the output from curl itself to the output destination in the config file. Instead it sends both the output from -w and the output from curl to the redirected stdout intended for the -w data only. Is there some way to set curl so that it sends -w output to one file in append mode, and uses the file specified in the config file for the output of the curl command?
I know it is reading the URL from the config file, because that is the source of the URLs. I know some of the URLs are written to files specified in the config file.
Could it be that the behavior of curl if it fails to get the filename/file path from the config file it defaults back to stdout? If that's the case, then could there be some issue with the filename parsing (maybe my filenames are too long or being ignored? I have long file paths and long file names in the config file) Is there a maximum length for a filename or maximum number of directories deep for the path?
I really appreciate the help as this has me stumped! All the hard work everyone has done in the 8.x branch has made this perform substantially better than the previous versions...if I can just get the kinks worked out in my process :)
On Monday, June 12, 2023 at 05:12:05 PM EDT, Daniel Stenberg <daniel_at_haxx.se> wrote:
On Mon, 12 Jun 2023, TD via curl-users wrote:
> Using Curl 8.10 compiled with openSSL on Ubuntu 20.04.
You probably mean 8.1.0
> I know curl has threading options
You probably mean the parallel option. curl does many simultaneous transfers
in the same thread.
> If I use GNU curl, the output from the -w gets added to the output from the
> curl command and it all ends up in one file.
Because you send them both to stdout?
> cat $ListOfConfigFilesToPass |parallel -P3000 curl -s -w "$listOfOptions"
> --doh-url $DOH_URL -A $AGENT_STRING --connect-timeout 15 --max-time 60 -k -L
> --config {} >>$outputFileForStats 2>/dev/null It seems like there should be
> 3 distinct outputs from this:---stdout (the output of -w) ---stderr (blank
> because the -s option should suppress it, and anything else is being sent to
> /dev/null)---the output file specified in the config file.
>
> Am I interpreting this correctly?
Not exactly. The -w output goes to stdout and since you don't use --output,
the normal data is also output to stdout.
-- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2023-06-13