cURL / Mailing Lists / curl-users / Single Mail

curl-users

Re: Smart strategy to write million of files in OSX with bash curl?

From: Rodrigo Zanatta Silva <rodrigozanattasilva_at_gmail.com>
Date: Fri, 16 Jan 2015 22:04:44 -0200

>I would approach the problem by downloading them as individual files using
the
>-O option (as you presumably are doing currently) with the additional step
of
>having curl print the file names using the -w option to stdout. I'd then
pipe
>stdout to a separate process (like zip or another shell script) which
would use
>the list of file names to consolidate the individual files into a single
large file.

Yeah... I understand what you say and the -w option. I will try using the
database approach. Zip is one solution too... but... I think the database
is a best option.

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2015-01-17