I'm having a problem with truncated files retrieved via FTP in ASCII mode
using cURL.

The FTP server is returning the wrong SIZE for the file because it doesn't
take the mode into account. (I think this is a common FTP server bug.)

This causes cURL to read too few bytes, although they are being sent by the
server.

Here's a 21 byte test file to illustrate the problem:

   abc[lf]

   def[lf]

   ghi[lf]

   jkl[lf]

   mno[lf]

   [lf]

The FTP server reports the SIZE as 21 bytes regardless of the mode but sends
27 bytes on the ASCII FTP transfer:

   abc[cr][lf]

   def[cr][lf]

   ghi[cr][lf]

   jkl[cr][lf]

   mno[cr][lf]

   [cr][lf]

Because the size is wrong, the file is truncated after being transferred:

   abc[lf]

   def[lf]

   ghi[lf]

   jkl[lf]

   m

In cURL version 7.15.5 this same transfer works, although there's a warning
in the trace:

== Info: transfer closed with -6 bytes remaining to read

== Info: Received only partial file: 27 bytes

I dug into the differences and noticed that in v7.16.0 the size parameter of
the Curl_read call in Curl_readwrite (now readwrite_data in 7.19.7) was
changed to use bytestoread instead of buffersize:

       /* This is where we loop until we have read everything there is to

          read or we get a EWOULDBLOCK */

       do {

         size_t buffersize = data->set.buffer_size?

          data->set.buffer_size:BUFSIZE;

+ size_t bytestoread = buffersize;

+ int readrc;

+

+ if (k->size != -1 && !k->header)

+ bytestoread = (size_t)(k->size - k->bytecount);

         /* receive data from the network! */

- int readrc = Curl_read(conn, conn->sockfd, k->buf, buffersize,
&nread);

+ readrc = Curl_read(conn, conn->sockfd, k->buf, bytestoread,
&nread);

This means that Curl_read is being called with a size of 21 as reported by
the buggy server instead of buffersize's 16384 bytes.

So only 21 of the sent 27 bytes are being read.

I can kludge the ftp_state_get_resp function in ftp.c to get back to the old
behavior:

       if(result)

         return result;

     }

+ if((instate != FTP_LIST) && (data->set.prefer_ascii))

+ size = -1; /* kludge for servers that misstate file size in ASCII
mode */

   if(size > data->req.maxdownload && data->req.maxdownload > 0)

     size = data->req.size = data->req.maxdownload;

   infof(data, "Maxdownload = %" FORMAT_OFF_T "\n", data->req.maxdownload);

   if(instate != FTP_LIST)

     infof(data, "Getting file with size: %" FORMAT_OFF_T "\n", size);

   /* FTP download: */

   result=Curl_setup_transfer(conn, SECONDARYSOCKET, size, FALSE,

                              ftp->bytecountp,

                              -1, NULL); /* no upload here */

But there must be a better way to deal with this...

-David McCreedy

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2010-01-14