Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Issues with output
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: TD via curl-users <curl-users_at_lists.haxx.se>
Date: Mon, 12 Jun 2023 20:14:41 +0000 (UTC)
I am not sure if this is a bug, or operator error, so I am asking here first :)
Using Curl 8.10 compiled with openSSL on Ubuntu 20.04.
What I am trying to do is to use curl to retrieve a list of websites specified in a config file, with GNU Parallel on the command line to start multiple instances of curl at the same time using different config files, outputting the data for each website to the file specified in the config file, and outputting the stats from -w to a single stats file.
I know curl has threading options, but I had some issues getting them to work with DOH (I couldn't figure out the memory usage, I think due to the addition of a resolver cache to curl's process memory, but I haven't been able to validate that), so I reverted to this method. What I cannot figure out is why the output is getting mixed up when I use GNU parallel.
The issue is that when I use GNU parallel to pass the config file on the command line, my output gets all messed up. The stats file has HTML output in it, and the output files don't have the correct data from the correct websites in them.
Each line in the config file contains a url, and a unique path to output the data from that URL.
This works fine if I run a single instance of curl with the config specified on the command line and don't use GNU parallel to pass the config file.
If I use GNU curl, the output from the -w gets added to the output from the curl command and it all ends up in one file. Because I have multiple instances running at once, the data in that file is all mixed together and not useful.
my command line is:
cat $ListOfConfigFilesToPass |parallel -P3000 curl -s -w "$listOfOptions" --doh-url $DOH_URL -A $AGENT_STRING --connect-timeout 15 --max-time 60 -k -L --config {} >>$outputFileForStats 2>/dev/null
It seems like there should be 3 distinct outputs from this:---stdout (the output of -w)
---stderr (blank because the -s option should suppress it, and anything else is being sent to /dev/null)---the output file specified in the config file.
Am I interpreting this correctly?
Is there a better way to do this without using threads?
Thanks in advance for the assistance.TD
Date: Mon, 12 Jun 2023 20:14:41 +0000 (UTC)
I am not sure if this is a bug, or operator error, so I am asking here first :)
Using Curl 8.10 compiled with openSSL on Ubuntu 20.04.
What I am trying to do is to use curl to retrieve a list of websites specified in a config file, with GNU Parallel on the command line to start multiple instances of curl at the same time using different config files, outputting the data for each website to the file specified in the config file, and outputting the stats from -w to a single stats file.
I know curl has threading options, but I had some issues getting them to work with DOH (I couldn't figure out the memory usage, I think due to the addition of a resolver cache to curl's process memory, but I haven't been able to validate that), so I reverted to this method. What I cannot figure out is why the output is getting mixed up when I use GNU parallel.
The issue is that when I use GNU parallel to pass the config file on the command line, my output gets all messed up. The stats file has HTML output in it, and the output files don't have the correct data from the correct websites in them.
Each line in the config file contains a url, and a unique path to output the data from that URL.
This works fine if I run a single instance of curl with the config specified on the command line and don't use GNU parallel to pass the config file.
If I use GNU curl, the output from the -w gets added to the output from the curl command and it all ends up in one file. Because I have multiple instances running at once, the data in that file is all mixed together and not useful.
my command line is:
cat $ListOfConfigFilesToPass |parallel -P3000 curl -s -w "$listOfOptions" --doh-url $DOH_URL -A $AGENT_STRING --connect-timeout 15 --max-time 60 -k -L --config {} >>$outputFileForStats 2>/dev/null
It seems like there should be 3 distinct outputs from this:---stdout (the output of -w)
---stderr (blank because the -s option should suppress it, and anything else is being sent to /dev/null)---the output file specified in the config file.
Am I interpreting this correctly?
Is there a better way to do this without using threads?
Thanks in advance for the assistance.TD
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2023-06-12