Re: feature request: expected payload size command-line flag

From: Timothe Litt <litt_at_acm.org>
Date: Wed, 2 Nov 2022 11:33:46 -0400

On 02-Nov-22 10:53, Dan Fandrich via curl-users wrote:
> On Wed, Nov 02, 2022 at 07:23:04AM +0000, Danny McClanahan via curl-users wrote:
>> In fact, I believe /every/ command line using --ignore-content-length would be able to make use of the --expected-filesize flag I'm proposing here; since that flag was deemed useful enough to add, and --expected-filesize seems usable for strictly /more/ use cases than --ignore-content-length
> While there may be some cases where the supplied Content-Length: is wrong,
> necessitating the use of --ignore-content-length, there needs to additionally
> be a source for the *correct* Content-Length: in order to use the new proposed
> flag. I can't think of many cases where both conditions would be true.
>
> Dan

I'm not a supporter of this proposal, but to be fair:  if it's for a
progress meter, the value doesn't have to be exact - or even very
close.    A progress meter that's off by 20% is probably reassuring
enough for most purposes.  (In most cases, seeing motion and a rate is
more reassuring than the exact % done or end time.) And, if you're doing
something repetitive - e.g. a new release of curl, or a database update,
the size of the last download is probably a good enough proxy for the
next one. Sometimes, on some file systems pre-allocation can be worthwhile.

That said, I don't see a lot of productive use from this proposal.  And
lots of opportunity for misunderstandings. Requests for more precise
approximations and methods for determining what the "expected" size
ought to be.  Think "expected size from reference file".  Or previous
session?  Or an astrologer.  Or...  And what happens when the expected
size is wrong?  Is it an error?  Is there a tolerance on the actual vs.
expected before declaring an error?  Does this lead to --expected-time
when the size isn't available? --expected-size=1.2GB+3%-0.5%?

The examples are warnings, not encouragement!  It's a pretty deep rabbit
hole to go down without some significant demand and use cases.

Since you get a callback as data arrives, anyone with advance knowledge
of a transfer size can easily produce their own meter.

There is a corresponding case for uploads - if uploading from a file,
the size is knowable.  But not if from a pipe (especially a pipe from a
compressor.)

I don't see a significant return on investment in either case.

With respect to --ignore-content-length -- that's useful because,
unfortunately, some servers lie.  One lie is the wrong length. Another
is the right length, but missing or extra data, which is a detectable
error.  Ignoring serves the useful purpose of allowing (some) downloads
from such servers.


Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.

-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2022-11-02