cURL
Haxx ad
libcurl

curl's project page on SourceForge.net

Sponsors:
Haxx

cURL > Mailing List > Monthly Index > Single Mail

curl-tracker mailing list Archives

[ curl-Bugs-1822405 ] curlftpfs hangs after reading files with no write permission

From: SourceForge.net <noreply_at_sourceforge.net>
Date: Fri, 02 Nov 2007 16:51:20 -0700

Bugs item #1822405, was opened at 2007-10-29 23:27
Message generated for change (Comment added) made by bagder
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1822405&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: libcurl
Group: hang
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: n.f. (norbertf)
Assigned to: Daniel Stenberg (bagder)
Summary: curlftpfs hangs after reading files with no write permission

Initial Comment:
i have already reported this problem to the curlftpfs bugzilla.

http://sourceforge.net/tracker/index.php?func=detail&aid=1809225&group_id=160565&atid=816357

i'm not sure if this is really a libcurl problem, but there is a suspicion that libcurl doesn't move into a working state after reading a file failed (because no read permissions for the ftp user).

if there is a possibility to reproduce this bug without curlftpfs please tell me and i will try (perhaps a peace of demo code, which also uses curl_multi).

----------------------------------------------------------------------

>Comment By: Daniel Stenberg (bagder)
Date: 2007-11-03 00:51

Message:
Logged In: YES
user_id=1110
Originator: NO

it doesn't matter, that patch is still fine but it seems it wasn't
relevant to this case

----------------------------------------------------------------------

Comment By: n.f. (norbertf)
Date: 2007-11-03 00:50

Message:
Logged In: YES
user_id=135573
Originator: YES

should i revert the previous patch?

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-11-03 00:35

Message:
Logged In: YES
user_id=1110
Originator: NO

Strike that, here's another patch to test:

diff -u -b -r1.152 multi.c
--- multi.c 27 Sep 2007 01:45:23 -0000 1.152
+++ multi.c 2 Nov 2007 23:34:29 -0000
@@ -1181,15 +1181,20 @@
 
         if(CURLE_OK == easy->result)
           easy->result = Curl_readwrite_init(easy->easy_conn);
- else
- /* Remove ourselves from the send pipeline */
- Curl_removeHandleFromPipeline(easy->easy_handle,
- easy->easy_conn->send_pipe);
+
+ /* No need to remove ourselves from the send pipeline here since
that
+ is done for us in Curl_done() */
 
         if(CURLE_OK == easy->result) {
           multistate(easy, CURLM_STATE_DO_DONE);
           result = CURLM_CALL_MULTI_PERFORM;
         }
+ else {
+ /* failure detected */
+ Curl_posttransfer(easy->easy_handle);
+ Curl_done(&easy->easy_conn, easy->result, FALSE);
+ disconnect_conn = TRUE;
+ }
       }
       break;
 

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-11-02 23:04

Message:
Logged In: YES
user_id=1110
Originator: NO

My suggested patch was an attempt to fix the problem with the not
disconnecting the data connection on that failure.

Doesn't Curl_protocol_connecting() return a non-zero value when this
occurs?

----------------------------------------------------------------------

Comment By: n.f. (norbertf)
Date: 2007-11-02 17:15

Message:
Logged In: YES
user_id=135573
Originator: YES

unfortunately it doesn't. but with adding lots of fprintf's to libcurl and
curlftpfs i understand the problem now. especially printing the client side
port when opening the data connection was interesting.

for files which return "Permission denied" the data connection is not
closed. therefore in my case there are three data connections to the same
data port (which should never happen). the server doesn't know to which one
to push the file data - he takes the first one which has been opened for
unreadable "messages.4". unfortunately curlftpfs/libcurl selects() on the
last data connection where it expects the file data from lall.txt, but
there is no data arriving on this connection.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-10-31 22:50

Message:
Logged In: YES
user_id=1110
Originator: NO

I believe I found a flaw thanks to your log. Can you check if this patch
improves things for you?

--- lib/multi.c 27 Sep 2007 01:45:23 -0000 1.152
+++ lib/multi.c 31 Oct 2007 21:49:00 -0000
@@ -1057,7 +1057,7 @@
       /* protocol-specific connect phase */
       easy->result = Curl_protocol_connecting(easy->easy_conn,
                                               &protocol_connect);
- if(protocol_connect) {
+ if((easy->result == CURLE_OK) && protocol_connect) {
         /* after the connect has completed, go WAITDO */
         multistate(easy, CURLM_STATE_WAITDO);
         result = CURLM_CALL_MULTI_PERFORM;

----------------------------------------------------------------------

Comment By: n.f. (norbertf)
Date: 2007-10-31 17:52

Message:
Logged In: YES
user_id=135573
Originator: YES

i checked with wireshark that the server sends lall.txt completely and
also seems to close the data socket (there is a FIN/ACK packet from the
server).

there is only one thread and the problem seems always be related to the
two previous RETRs which fail. it only happens when there are read
protected files on a server. i can read as many files as i want when they
all have read permissions - i can also read lall.txt as often as i want, it
only hangs in this paricular case.

i will send you the wireshark file to your email address.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-10-31 15:23

Message:
Logged In: YES
user_id=1110
Originator: NO

So what _does_ the server do when you do RETR on this file? Judging from
the log, it seems it says there's gonna be a file coming but then it
doesn't send it or possibly not the whole file and it doesn't seem to close
the connection.

You can possibly use wireshark to verify/check this.

Regarding the close on the previous connection or whatever, that's a
subject for a different thread/issue whatever and I won't address that in
this bug report.

----------------------------------------------------------------------

Comment By: n.f. (norbertf)
Date: 2007-10-31 13:18

Message:
Logged In: YES
user_id=135573
Originator: YES

ok - here is another log with the important parts and some curlftpfs
debugging output.

i don't understand why the data connection from a previous (failed) RETR
attempt is still open. i thought curl_multi_remove_handle() should close
it. or perhaps libcurl expects that the server closes the connection, but
proftpd doesn't...

File Added: curlftpfs3.log

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-10-31 10:14

Message:
Logged In: YES
user_id=1110
Originator: NO

First, line 428 is completely wrong on all libcurl versions before 7.17.0.
The curl_easy_setopt man page currently says:

  "NOTE: before 7.17.0 strings were not copied. Instead the user was
forced keep
  them available until libcurl no longer needed them."

If this mistake is done once, perhaps it is made more times at other
places...

And regarding EPSV vs PASV, they're more or less identical in the client
end. They're just two different commands with slightly difference
responses, but other than those details libcurl makes no difference at all
in how it acts and communicates using either approach.

Your log is hard to follow since it is multiple requests shown in a single
log. But yes, a data connection should always be closed after the transfer.

----------------------------------------------------------------------

Comment By: n.f. (norbertf)
Date: 2007-10-31 00:42

Message:
Logged In: YES
user_id=135573
Originator: YES

AFAIK there is no firewall between my workstation and my ftp server. They
are in the same private network.

I tried a ethereal dump. Below the conversation of the control channel.
Furthermore there are three TCP connections to data port 50098. Two are
empty and one pushes back the data of the readable file lall.txt. The weird
thing is that one of the empty data connections (the second one) remains
ESTABLIHED. It seems to be the one opened for the second read protected
file (messages.3).

I don't know the EPSV protocol. Shouldn't the ftp client close this
connection?

230 User XXXX logged in.
PWD
257 "/" is current directory.
CWD old_messages
250 CWD command successful.
EPSV
229 Entering Extended Passive Mode (|||50098|)
TYPE I
200 Type set to I
SIZE messages.4
213 20628
RETR messages.4
550 messages.4: Permission denied
EPSV
229 Entering Extended Passive Mode (|||50098|)
SIZE messages.3
213 10804
RETR messages.3
550 messages.3: Permission denied
EPSV
229 Entering Extended Passive Mode (|||50098|)
SIZE lall.txt
213 15
RETR lall.txt
150 Opening BINARY mode data connection for lall.txt (15 bytes)
226 Transfer complete.
------ (hangs) -----

i might try to write a test case which resembles the behavior of
curlftpfs. but it won't be easy. :-(
perhaps you could also have a look at the ftpfs_read_chunk() function at
line 392 of

http://curlftpfs.cvs.sourceforge.net/curlftpfs/curlftpfs/ftpfs.c?revision=1.25&view=markup

maybe there is something obviously wrong... i think the problem is that
the select (line 428) loop doesn't return when reading lall.txt.

thx
norbert

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-10-30 22:26

Message:
Logged In: YES
user_id=1110
Originator: NO

That "recommendation" is worth nothing here. Yeah, EPSV might cause
problems for people who use weird NATs/firewalls etc, but there too it
doesn't cause trouble to libcurl only to users of it or perhaps the entire
app.

Can you please do a plain libcurl app that repeats the problem against
your server and show it to us? As long as you need curlftpfs to repeat it,
we will continue to suspect the bug is outside of libcurl...

----------------------------------------------------------------------

Comment By: n.f. (norbertf)
Date: 2007-10-30 21:25

Message:
Logged In: YES
user_id=135573
Originator: YES

no. i didn't test PORT mode, because curlftpfs doesn't seem to have a
switch to force PORT mode.

what i can say is that the problem occurs in EPSV but not in PASV mode.

i also found this "recommendation":

http://www.ohloh.net/projects/7238/contributors/40480/commits/11711342

and yes, the problem occurs on every hit (on my own ftp server - a
proftpd-1.2.9-1.9). actually i have to try to read two files with no read
permission and then reading the third (not read protected file) hangs.

i just tried on an external FTP server - also with EPSV - but i can't
reproduce the problem there.

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2007-10-30 21:00

Message:
Logged In: YES
user_id=236775
Originator: NO

Are you saying the problem occurs in PORT mode? Because the log attached
to the other bug is also using passive mode, as is test case 546 (EPSV
mode, to be specific). Does this problem happen every single time this
condition is hit, or just sporadically?

----------------------------------------------------------------------

Comment By: n.f. (norbertf)
Date: 2007-10-30 20:32

Message:
Logged In: YES
user_id=135573
Originator: YES

i tried to investigate this further:

this behavior might indicate that this is a libcurl bug:
the problem goes away when i use PASV mode ( -o disable_epsv)

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-10-30 10:06

Message:
Logged In: YES
user_id=1110
Originator: NO

Since this doesn't seem to be a bug in libcurl...

----------------------------------------------------------------------

Comment By: n.f. (norbertf)
Date: 2007-10-30 01:18

Message:
Logged In: YES
user_id=135573
Originator: YES

thanks a lot, i will try your test-case as soon as possible.

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2007-10-30 00:51

Message:
Logged In: YES
user_id=236775
Originator: NO

The error code from the FTP server in both cases is 550, so from libcurl's
perspective it's identical. The difference with your case is that the SIZE
command succeeds, but libcurl handled that fine as well when I tried
changing the test case to match. It's possible that there are other
differences in how curlftpfs uses libcurl, but this test case at least
gives you a known-working baseline to which you can refer.

----------------------------------------------------------------------

Comment By: n.f. (norbertf)
Date: 2007-10-30 00:33

Message:
Logged In: YES
user_id=135573
Originator: YES

thanks, but i think that's not a file-not-found problem, but a
permission-denied problem.

> RETR messages.4
* FTP 0x80658d0 state change from RETR_SIZE to RETR
< 550 messages.4: Permission denied
* RETR response: 550
...
> RETR messages.3
* FTP 0x80658d0 state change from RETR_SIZE to RETR
< 550 messages.3: Permission denied
* RETR response: 550
...

have a look at curlftplog.txt, which i attached to the curlftpfs bug
report.

after trying to read messages.4 and messages.3 (which both have no read
permissions), reading messages.2 (for which i have read permissions)
hangs:

...
> RETR messages.2
* FTP 0x80658d0 state change from RETR_SIZE to RETR
< 150 Opening BINARY mode data connection for messages.2 (84257 bytes)
* Maxdownload = -1
* Getting file with size: 84257
* FTP 0x80658d0 state change from RETR to STOP
* DO-MORE phase ends with 0
* STATE: DO_MORE => DO_DONE handle 0x808e3e4; (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x808e3e4; (connection #0)
* Conn 0 recv pipe 1 inuse 0 athead 1
* STATE: WAITPERFORM => PERFORM handle 0x808e3e4; (connection #0)

(hangs)

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2007-10-30 00:01

Message:
Logged In: YES
user_id=236775
Originator: NO

I've created test case 546 that tests a file-not-found FTP transfer
followed by a found one using the multi interface, and it passes fine.

----------------------------------------------------------------------

Comment By: n.f. (norbertf)
Date: 2007-10-29 23:40

Message:
Logged In: YES
user_id=135573
Originator: YES

sorry - the title of this bug report should be:
curlftpfs hangs after reading files with no READ permission

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1822405&group_id=976
Received on 2007-11-03

These mail archives are generated by hypermail.

donate! Page updated November 12, 2010.
web site info

File upload with ASP.NET