For what it's worth, I thought I'd post an update to our problem with
Microsoft FTP servers. We're getting closer and hope to get some
additional feedback on how we can overcome this last hurdle.

We have determined that for whatever reason, the FTP server in Windows
2000 (IIS 5) in our environment will intermittently drop connections.
The IIS 5 FTP server also deletes partial uploads when a transfer does
not complete, so the APPE command will not work correctly. Win2K3 does
not delete partial uploads.

We are now seeing a similar issue when uploading to IIS6/Win2K3. At the
end of *some* file transfers we do not see the typical 226/QUIT/221
message sequence, but a similar connection RST issue and the resulting
file is truncated. It doesn't happen on all files, but it happens
consistently on files of certain sizes.

Rob Lievaart suggested a scenario where a command connection request may
be getting to the server before the data connection upload/closure
sequence completes. Though we don't see cURL checking SIZE at the end
of the transfer, we felt that there is a good chance that we are seeing
a similar issue.

We have 3 different scenarios and 2 duplicate the problem. In both
failure cases after the transfer completes, the data connection is
closed and then immediately the curl cleanup functions are called. In
the case that seems to always work, we keep the curl handle around to be
reused, thus the cleanup is delayed significantly after the transfer
completes. In the failure cases, if we introduce a hard-coded sleep
after the data connection is closed but before curl is cleaned up, all
file transfers succeed!!

THE THEORY: It seems that if we close the data connection and clean up
curl too quickly, the MS FTP server is still processing or flushing the
last bits of the data stream to disk when the command connection is
closed. The server thinks there was an error so it sends a RST and
aborts, causing the file to be truncated, thought curl thinks everything
is happy.

We are now trying to figure out how to know that the data connection has
been cleanly closed before we attempt the cleanup. I assume (based on
incomplete knowledge of the protocol mechanics) that once the data
connection is closed and the server finishes whatever it does on its end
to complete the transfer, it should send a 226 back to the client.

THE BIG QUESTION: What is the mechanism to watch and wait for the 226 -
Transfer Complete message from the server before initiating the cleanup?
I can not find any examples/docs of how this might be done as it seems
that once initiated, curl handles everything internally and never
exposes the 226 or other server messages.

Any suggestions/helps/links are much appreciated!! Thanks.

-Jeff

-----Original Message-----
From: curl-library-bounces_at_cool.haxx.se
[mailto:curl-library-bounces_at_cool.haxx.se] On Behalf Of Jeff Vincent
Sent: Tuesday, October 30, 2007 9:24 AM
To: libcurl development
Subject: RE: libcurl and Microsoft IIS FTP server

Thanks, I have informed the developer.

The file I have been using recently is 270MB. I have tried 1 GB and
larger files many weeks ago but needed to get a smaller file so I could
get through debugging my tests more quickly. I'll retry some larger
files soon.

-Jeff

-----Original Message-----
From: curl-library-bounces_at_cool.haxx.se
[mailto:curl-library-bounces_at_cool.haxx.se] On Behalf Of Rob Lievaart
Sent: Tuesday, October 30, 2007 5:29 AM
To: libcurl development
Subject: RE: libcurl and Microsoft IIS FTP server

Hi,
 
> > I'm new to this list and have tried searching the archives for my
issue but
> > without success. I am debugging an app (that I didn't write) that
uses
> > libcurl for FTP support. When copying against a Windows 2000 Server
running
> > the latest service packs (sp4) and using the MS IIS FTP server I am
getting
> > some weird behavior. I can't find anything to tell me there is a
problem
> > with the MS FTP server or that it behaves differently than other FTP
server
> > implementations.

Are the files perhaps larger then 2 Gigabyte? I have seen similar
behavior with an older libcurl versions and the MS-FTP server. The
application passed a 64 bit filesize to CURLOPT_INFILESIZE (Which should
be 32 bits). All seems well but when curl checks the filesize at the end
of the transfer, it finds
an error and closes the connection. The closing of the command
connection
reaches the server before the last packets of the actual file transfer.
When the
command connection closes MS-FTP cleans up the file.

My $0.02,

Rob Lievaart
Received on 2007-12-06