curl-users
Re: Smart strategy to write million of files in OSX with bash curl?
Date: Fri, 16 Jan 2015 22:04:44 -0200
>I would approach the problem by downloading them as individual files using
the
>-O option (as you presumably are doing currently) with the additional step
of
>having curl print the file names using the -w option to stdout. I'd then
pipe
>stdout to a separate process (like zip or another shell script) which
would use
>the list of file names to consolidate the individual files into a single
large file.
Yeah... I understand what you say and the -w option. I will try using the
database approach. Zip is one solution too... but... I think the database
is a best option.
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2015-01-17