Re: How to use curl -Z (--parallel) effectively?

From: Pierre Brico via curl-users <curl-users_at_lists.haxx.se>
Date: Sun, 6 Mar 2022 18:40:49 +0100

Hi Jacques,

You can indeed use the "--config" file to specify a file containing the
list of URLs to be downloaded. The syntax is:
url="YOU_URL"
output="OUTPUT_FILE_WITH_FOLDER"
url=...
output=...
....

You can add an unlimited number of files to download. The command line to
launch the download process is:
curl --parallel --parallel-immediate --parallel-max 5 --config
[FILE_WITH_LIST]

Here is a fragment of bash script to build the list and launch the download:

echo "Generating list of chunks to be downloaded"

exec 7>$FFMPEG_LIST
exec 8>$CURL_URLS

cat $CHUNK_FILE | grep -v "^#" | while read chunk; do
  echo "url=\"$BASE_URL/$chunk\"" >&8
  echo "output=\"video/${chunk##*/}\"" >&8
  echo "file video/${chunk##*/}" >&7
done

# close both output files
exec 7</dev/null
exec 8</dev/null

echo "Start to download chunks"
curl --parallel --parallel-immediate --parallel-max 5 --config $CURL_URLS
if [ $? -ne 0 ]; then
        echo "Couldn't download chunks"
        exit 4
fi

echo "Merging all downloaded chunks to recreate the movie"
$FFMPEG_DIR/ffmpeg.exe -f concat -i $FFMPEG_LIST -acodec copy -vcodec copy
$MERGE_OUTPUT

I use this code to download video chunks and then merge all of them with
ffmpeg to build the final video.

I hope this help,
Pierre

On Sun, Mar 6, 2022 at 1:31 PM jacques granduel via curl-users <
curl-users_at_lists.haxx.se> wrote:

> Hi Curl Community,
>
> I have posted 2 questions on StackOverFlow
> <https://stackoverflow.com/questions/71244217/how-to-use-curl-z-parallel-effectively>
> as I thought I would get a very quick answer but didn't get any! I would
> like to get an answer anyway, for me and SOF users, if somehome come over
> it. Sorry in advance for this double posting.
> Here's my question:
>
> I need to download thousands of files with *curl*. I know how to
> parallelize with xargs -Pn (or gnu parallel) but I've just discovered
> curl itself can parallelize downloads with the argument -Z|--parallel
> introduced in *curl-7.66* (see curl-goez-parallel
> <https://daniel.haxx.se/blog/2019/07/22/curl-goez-parallel/>) which might
> be cleaner or easier to share. I need to use -o|--output option and
> --create-dirs. URLs need to be *percent-encoded*, the URL path becoming
> the folder path which also need to be escaped as path can contain single
> quotes, spaces, and usual suspects (hence -O option is not safe and -OJ
> option doesn't help). If I understand well, curl command should be build
> like so:
>
> curl -Z -o path/to/file1 http://site/path/to/file1 -o path/to/file2 http://site/path/to/file2 [-o path/to/file3 http://site/path/to/file3, etc.]
>
> This works indeed, but what's the best way to deal with thousands URLS.
> Can a config file used with -K config be useful? what if the -o
> path/to/file_x http://site/path/to/file_x is the output of another
> program? I haven't found any way to record commands in a file, one command
> per line, say.
>
> Thanks in advance if you can give me any tip!
>
> Best regards
> --
> Unsubscribe: https://lists.haxx.se/listinfo/curl-users
> Etiquette: https://curl.haxx.se/mail/etiquette.html
>

-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-users
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Received on 2022-03-06