On Thu, 6 Dec 2007, Jeff Vincent wrote:

> We are now seeing a similar issue when uploading to IIS6/Win2K3.

And this is still only libcurl that shows this behaviour? What libcurl version
are you using now? Have you compared the ftp command sequence with a working
ftp client with the particular sequence that libcurl uses?

> At the end of *some* file transfers we do not see the typical 226/QUIT/221
> message sequence, but a similar connection RST issue and the resulting file
> is truncated. It doesn't happen on all files, but it happens consistently
> on files of certain sizes.

libcurl waits for the 226 response when it still thinks everything is fine, it
doesn't if it already has deemed the transfer to have failed. See
Curl_ftp_done() in lib/ftp.c.

> Rob Lievaart suggested a scenario where a command connection request may be
> getting to the server before the data connection upload/closure sequence
> completes.

I don't see how this can happen, and if it does happen it is a major server
bug that I can't see what a client can do to avoid.

> Though we don't see cURL checking SIZE at the end of the transfer, we felt
> that there is a good chance that we are seeing a similar issue.

No it doesn't check SIZE, why would it? libcurl knows how much data it sent
and if it sent the entire file without problems, it just needs to wait for the
226 (or 250) to know the file transfer was fine.

> In the case that seems to always work, we keep the curl handle around to be
> reused, thus the cleanup is delayed significantly after the transfer
> completes. In the failure cases, if we introduce a hard-coded sleep after
> the data connection is closed but before curl is cleaned up, all file
> transfers succeed!!

This somehow implies that Curl_ftp_done() is buggy or that its 60 second
timeout is too short?

> THE THEORY: It seems that if we close the data connection and clean up curl
> too quickly, the MS FTP server is still processing or flushing the last bits
> of the data stream to disk when the command connection is closed. The
> server thinks there was an error so it sends a RST and aborts, causing the
> file to be truncated, thought curl thinks everything is happy.

I don't see how this theory matches how the code is written-

First, the data connection is *OF COURSE* closed when the transfer is complete
as that's how a client signals that the transfer is complete. There's no other
way to do this - it just can't be wrong. See RFC959.

Then, libcurl _does_ wait for the server to tell that the operation is fine
(or not) before it continues and if the server does tell it things are fine
before it is then the server is buggy. If the server doesn't say that and
libcurl still continues (but the 226 comes after yet a while) then our
hardcoded 60 second timeout in Curl_ftp_done() is too strict and should be
prolonged.

> THE BIG QUESTION: What is the mechanism to watch and wait for the 226 -
> Transfer Complete message from the server before initiating the cleanup?

libcurl always does that.

> I can not find any examples/docs of how this might be done as it seems
> that once initiated, curl handles everything internally and never
> exposes the 226 or other server messages.

libcurl exposes the 226 response the same way as all other respones, in header
callbacks. All this that you talk about are details that just should work,
with no particular treatment required from an application so I don't see why
any particular exposure of the 226 is necessary.

> Any suggestions/helps/links are much appreciated!! Thanks.

Oh, and please don't top-post, it makes it totally confusing to read quotes in
mails in mixed (or just reversed) order.

-- 
  Commercial curl and libcurl Technical Support: http://haxx.se/curl.html