On Wed, 19 Apr 2006, Daniel Stenberg wrote:

>Doesn't the "Almost all" and "some" of the paragraph [below] make such a
>calculation and assumption rather error-prone?

Yes, it is error-prone. And I'd be happy with a more reliable method of
handling this but I don't know that there is one since we're dependent on
the FTP server giving us the right SIZE response.

But it also implies that the current code has some of the same
vulnerabilities with line end handling...
For example, let's transfer a file from Linux using a server that complies
with the RFC959 and converts LFs to CRLFs on text/ascii mode transfers, but
doesn't adjust the SIZE reply based on transfer mode.
This example file is 100 bytes including 10 LFs.
The server (under)reports a SIZE of 100 bytes.
It converts LFs to CRLFs and sends 110 bytes.
If 10 bytes get lost, Curl receives 100 bytes which matches the SIZE
response and all is thought to be well.

Granted, the message has to lose the same number of bytes as the file has
LFs, but that's true of the proposed line end handling code too.

And if the server doesn't support the optional SIZE command we're completely
in the dark about file size.

A real concern here is that Curl closes the connection if the (greater than
zero) SIZE doesn't match the number of bytes FTPed.
That would be almost all text/ascii transfers. Not a desirable outcome.

We could make inbound and outbound line end translations optional,
controlled by a new CURLOPTion for FTP.
That way the caller would be explicitly aware of it and the potential for
data loss (if so documented).
Actually, I'd lean towards a new CURLOPTion just so line end handling is
more flexible instead of saying Platform X always does line end translation
on text/ascii transfers.

I'm open to suggestions.

-David

>>On Wed, 12 Apr 2006, David McCreedy wrote:
>>1) Almost all FTP servers I've tried give the same SIZE regardless of the
>>mode (bin vs ascii) of the download. (Some servers do go to the extra
>>trouble of calculating the SIZE accurately for ascii mode transfers.) For
>>example, let's use an FTP server on a platform that uses LF for the
>>end-of-line marker. If the file is 50 bytes long (natively) with 10
>>lineends, the SIZE will usually be given as 50 bytes. That's correct for
>>bin mode. But for ASCII mode those LFs become CRLFs and we get 60 bytes
>>from the server. So we get a "partial" file with -10 bytes left to read.
>>I've tracked the number of CRLFs Curl converted so we can try to determine
>>if the size difference was really a partial file or just lineend related.
Received on 2006-04-20