Re: Issues with output

From: TD via curl-users <curl-users_at_lists.haxx.se>
Date: Sat, 17 Jun 2023 00:02:21 +0000 (UTC)

 

    On Tuesday, June 13, 2023 at 03:26:37 AM EDT, Ray Satiro via curl-users <curl-users_at_lists.haxx.se> wrote:
 
 On 6/12/2023 7:08 PM, TD via curl-users wrote:
> The config file has an output file specified as part of the config
> file, so I thought that all the output for the site would be sent to
> the file, and the output from -w with the stats would then go to
> stdout, which I redirect in append mode. This works on the command
> line with a single curl command and the config file set in the command
> line. It also works with multiple curl commands, if I use the -o
> option on the command line set with a variable (-o $site.out), and
> then redirect the stdout to a second file for the stats.


Please don't top-post [1], it makes the conversation harder to follow.

Possibly your configuration file is not in the right format [2].

Try this:

~~~
rm list

for x in {a..c}; do
cat << EOF > $x.cfg
url = "http://mockbin.com/echo"
data = "data_$x\n"
output = "$x.out"
EOF
echo $x.cfg >> list
done

cat list | parallel -P1 curl --config {}

for x in {a..c}; do cat $x.out; done
~~~

You should see:
data_a
data_b
data_c

If you get the above results and have followed the format please give us
a minimal self contained example that we can use to reproduce.



Thank you for the assistance.
I apologize for top posting. I will not do it again.I tried the test above, and it demonstrated the expected behavior.
I tried 2 different versions of curl 8.x and got the same result.
curl 8.1.2 (x86_64-pc-linux-gnu) libcurl/8.1.2 OpenSSL/1.1.1f zlib/1.2.11
Release-Date: 2023-05-30
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe TLS-SRP UnixSockets


also tried with this older version:

curl 8.0.1 (x86_64-pc-linux-gnu) libcurl/8.0.1 OpenSSL/1.1.1f zlib/1.2.11
Release-Date: 2023-03-20Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe TLS-SRP UnixSockets

 
Command line:
echo "/data2/domains/june-8-top-xyz/testconfig3"  |parallel -P1  curl-8/bin/curl  -s -w "time_namelookup: %{time_namelookup}, time_connect: %{time_connect}, time_appconnect: %{time_appconnect}, time_pretransfer: %{time_pretransfer}, time_redirect: %{time_redirect}, time_starttransfer: %{time_starttransfer}, time_total: %{time_total}\n"  --doh-url https://cloudflare-dns.com/dns-query  -A "Mozilla/5.0 \(Windows NT 10.0\; Win64\; x64\) AppleWebKit/537.36 \(KHTML, like Gecko\) Chrome/100.0.4896.60 Safari/537.36" --connect-timeout 15 --max-time 60 -k -L --config {} >> /data1/domains/tbad7.out

 

config file looks like this (verified in vi that all lines end with a newline immediately after the quote, no spaces)
url = http://heezytuis.xyz
output = "./output/test1/http/site/heezytuis.xyz..out-http"
url = http://www.heezytuis.xyz
output = "./output/test1/http/www/www-heezytuis.xyz..out-http"
url = https://heezytuis.xyz
output = "./output/test1/https/site/heezytuis.xyz..out-https"
url = https://www.heezytuis.xyz
output = "./output/test1/https/www/www-heezytuis.xyz..out-https"
url = https://d.heezytuis.xyz/?jp=1
output = "./output/test1/https/d-site/d-heezytuis.xyz..out-https"
After some additional testing, I managed to get it to work with xargs -I{} -Pxx instead of GNU parallel.

I have been unable to determine why GNU parallel didn't work.
thanks.
TD
   
   -

  

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2023-06-17