On Mon, Feb 03, 2014 at 12:55:53PM +0100, Manfred Schwarb wrote:
> while doing parallel downloads with curl, I found that the resulting
> files are somewhat fragmented. I.e. doing
>
> for ((i=1; i<20; i++)); do
> curl -o $i.txt http://server/$i.txt &
> done
>
> So it seems curl does not fallocate the resulting files. Is there a
> possibility to achieve this? Or are there plans to add a fallocate
> option to curl?

I haven't heard anyone suggest it so far. There should be very little down side
to adding a call to Linux's fallocate(2) with FALLOC_FL_KEEP_SIZE when the size
is known. That call preallocates space but keeps the reported file size the
same--only when more data is appended to the file does it appear to grow in
size. The primary down side is (naturally) that the space is allocated at once,
so curl's behaviour w.r.t. handling out of disk space errors would be slightly
different. Plus, I don't think fallocated-space is automatically freed if a
download is aborted or the final file size is less than originally expected,
which would result in an unexplained loss of disk space. Another down side is
that this call is Linux-specific.

The posix_fallocate behaviour is quite different in that the reported file size
reaches the maximum right after the call. That makes download resumption
impossible as there's no reliable way to find out how much of the file has been
downloaded. It's also impossible in the general case to tell post facto whether
the file was downloaded successfully.

It's probably reasonable to unconditionally add an fallocate call to curl when
the size is known, but if only posix_fallocate is available I'd be against
calling it without a command-line option.

>>> Dan
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2014-02-03