Parallel download issue

Hi,

I wrote a long time ago a script to download videos. Some of them
(m3u8) are cut into chunks. So, the script downloads all chunks and
then merges them together with the excellent tool "ffmpeg" to create
the final video.

Reading the new features of curl, I've decided to improve my script to
use the parallel download which is an awesome feature. The throughput
jumps from 20Mbits/sec to 130Mbits/sec (which is my Internet bandwidth
limit) with just a parallel factor of 5.

My problem is that after some time, all received chunks are empty. For
example, for a video with 1800 chunks for a total space of 1GB, curl
downloads 922 chunks correctly and the rest is empty. Return code of
curl command is 0.

-rw-r--r-- 1 Pierre None 584304 Apr 28 17:47 1_vs0_00918.ts
-rw-r--r-- 1 Pierre None 585808 Apr 28 17:47 1_vs0_00919.ts
-rw-r--r-- 1 Pierre None 599156 Apr 28 17:47 1_vs0_00920.ts
-rw-r--r-- 1 Pierre None 573588 Apr 28 17:47 1_vs0_00921.ts
-rw-r--r-- 1 Pierre None 529972 Apr 28 17:47 1_vs0_00922.ts
-rw-r--r-- 1 Pierre None 0 Apr 28 17:47 1_vs0_00923.ts
-rw-r--r-- 1 Pierre None 0 Apr 28 17:47 1_vs0_00924.ts
-rw-r--r-- 1 Pierre None 0 Apr 28 17:47 1_vs0_00925.ts
-rw-r--r-- 1 Pierre None 0 Apr 28 17:47 1_vs0_00926.ts
-rw-r--r-- 1 Pierre None 0 Apr 28 17:47 1_vs0_00927.ts
-rw-r--r-- 1 Pierre None 0 Apr 28 17:47 1_vs0_00928.ts
-rw-r--r-- 1 Pierre None 0 Apr 28 17:47 1_vs0_00929.ts

I've tried several times and the limit oscillates between 900 and 950
files. If I restart the download from the last good chunk, curl
downloads around 400 chunks and then starts again to generate empty
files.
As chunks are downloaded from a remote server which I don't manage, I
can't see the server log files but I just wonder if someone else
already has tried this features with similar volumes.
How can I debug the problem (see if connections are broken or see HTTP
return code)?

Here is the command line:
curl -k -K curlList.txt --parallel --parallel-immediate --parallel-max 5

and a fragment of curlList.txt is (the URL was modified because it was
containing personal IDs):

url = https://xxxx/segments/1_vs0_00009.ts
output = video/1_vs0_00009.ts
url = https://xxxx/segments/1_vs0_00010.ts
output = video/1_vs0_00010.ts
url = https://xxxx/segments/1_vs0_00011.ts
output = video/1_vs0_00011.ts

Thanks,
Pierre
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2020-04-28