Hi all,

I have some trouble uploading a bunch of files to the same server using the FTP protocol.

Scenario: I am using a multi handle, having two to four easy handles added to it and a lot of queued files to upload, all to the same server / base URL. The basic idea is to think of each of the easy handles as a separate connection to the same server, so if I have two easy handles, I have two parallel "channels" to the server. The easy handles get populated with the transfer credentials for the first files to upload and are added to the multi handle. Then the following multi interface loop (pseudocode) gets executed:

----------------------------------<snip>----------------------------------

OsFileVector files; // std::vector contains paths of files to upload

OsFileVector::const_iterator file = files.begin();
bool result = true;

int running_handles;
CURLMcode multiResult;
while( result )
{
        // call curl_multi_perform, as long as it wants to be called:
        //
        while( ( multiResult = curl_multi_perform( curlMulti, &running_handles ) ) == CURLM_CALL_MULTI_PERFORM ) ;

        // now process waiting easy handle messages:
        //
        int num_messages;
        CURLMsg* curlMsg = curl_multi_info_read( curlMulti, &num_messages );
        while( curlMsg ) {
                if( curlMsg->msg == CURLMSG_DONE )
                {
                        CURL* easy = curlMsg->easy_handle;
                        CURLcode easyResult = curlMsg->data.result;
                        curl_multi_remove_handle( curlMulti, easy );
                        if( ( easyResult != CURLE_URL_MALFORMAT ) && ( easyResult != CURLE_OK ) )
                        {
                                // serious error occured.
                                result = false; // <---------- Active mode FTP ends up here with CURLE_FTP_PORT_FAILED
                                break;
                        }
                        // prepare and resurrect easy handle for next transfer:
                        if( file != files.end() )
                        {
                                curl_easy_setopt( easy, CURLOPT_URL , url );
                                curl_easy_setopt( easy, CURLOPT_READDATA , readData );
                                curl_multi_add_handle( curlMulti, easy );
                                file++;
                                running_handles++;
                        }
                }
                curlMsg = curl_multi_info_read( curlMulti, &num_messages );
        }

        if( !running_handles )
                break;

        // get curl timeout setting:
        //
        long timeout; // millisecs
        curl_multi_timeout( curlMulti, &timeout );
        if( timeout < 0 )
                timeout = DEFAULT_TIMEOUT_MILLISECS; // 250ms
        if( timeout > MAX_TIMEOUT_MILLISECS ) // 1000ms
                timeout = MAX_TIMEOUT_MILLISECS;

        // check socket action:
        //
        int fdMax;
        fd_set fdRead, fdWrite, fdException;
        FD_ZERO( &fdRead );
        FD_ZERO( &fdWrite );
        FD_ZERO( &fdException );
        curl_multi_fdset( curlMulti, &fdRead, &fdWrite, &fdException, &fdMax );
        if( fdMax != -1 )
        {
                // socket stuff to do:
                ldiv_t t = ldiv( timeout, 1000L );
                timeval tv;
                tv.tv_sec = t.quot;
                tv.tv_usec = t.rem * 1000;
                int select_result = select( fdMax + 1, &fdRead, &fdWrite, &fdException, &tv );
                if( select_result == SOCKET_ERROR )
                {
                        result = false;
                        break; // error
                }
        }
        else
        {
                // Nothing to do on the sockets.
                // Be nice to other threads and end our time slice here.
                yield();
        }

}

----------------------------------</snip>----------------------------------

Inside the loop, after curl_multi_perform() has been called repeatedly, I use curl_multi_info_read() to check for easy handles that are marked as "done" and detach those easy handles with curl_multi_remove_handle(). If there are files left to transfer, If re-populate the detached easy handle and call curl_multi_add_handle(), to add the easy handle to the multi handle again.

If I use only one easy handle for the multi handle, everything works fine with either active mode or passive mode FTP. The problem only occurs, if I use two or more easy handles.

The problem is, that after transferring some files (it's different for every run), either curl_multi_info_read() returns CURLE_FTP_PORT_FAILED for the particular transfer (that's for active mode FTP) or some of the easy handles simply stuck and sometimes returns CURLE_RECV_ERROR after several minutes (that's for passive mode FTP).

I obtained WireShark logs for the erroneous channel for both cases.

Active mode FTP:
----------------------------------<snip>----------------------------------
...etc...
PORT 1,2,3,4,5,253
200 PORT command successful
STOR adobe_labs_deliverable_01.cfm
150 Opening BINARY mode data connection for adobe_labs_deliverable_01.cfm
226 Transfer complete.
PORT 1,2,3,4,5,255
200 PORT command successful
STOR AJAXScope.html
150 Opening BINARY mode data connection for AJAXScope.html
PORT 1,2,3,4,6,0
226 Transfer complete. // 226 Result of previous transfer?? This leads to CURLE_FTP_PORT_FAILED
200 PORT command successful // 200 Result for PORT command
PORT 1,2,3,4,6,1 // another PORT command issued?? Possibly for the next transfer??
200 PORT command successful
STOR ajax_authoring_impl.html
150 Opening BINARY mode data connection for ajax_authoring_impl.html
----------------------------------</snip>----------------------------------

Passive mode FTP:
----------------------------------<snip>----------------------------------
PASV
227 Entering Passive Mode (1,2,3,4,225,58).
STOR ManageCSS_Spec.pdf
150 Opening BINARY mode data connection for ManageCSS_Spec.pdf
226 Transfer complete.
PASV
227 Entering Passive Mode (1,2,3,4,209,237).
STOR MMHTTPDB.cfm
150 Opening BINARY mode data connection for MMHTTPDB.cfm
226 Transfer complete.
PASV
227 Entering Passive Mode (1,2,3,4,211,100).
STOR open.png
150 Opening BINARY mode data connection for open.png
PASV // <--- channel stuck here, leading to CURLE_RECV_ERROR after several minutes

----------------------------------</snip>----------------------------------

Can anyone see something suspicious or obvious errors in this code?
Can I use the multi interface for this purpose in this way? Or is there a principal mistake in this approach?

The problem appears with different servers on different operating systems and transfer speeds. If I cannot get this fixed, it is a real show-stopper for my usage of the libcurl multi interface. So, any help is very much appreciated.

My environment is:

        MS-Windows XP
        MS VisualStudio 2005
        libcurl 7.19.0
        libssh2 0.19.0_CVS as of 10-10-2008
        c-ares 1.5.2
        ...

Cheers,
Andreas

--
Andreas Wurf <awurf_at_adobe.com>
Computer Scientist
Adobe Systems Engineering GmbH
Registergericht: Hamburg HRB 745 37, Geschäftsführer: Thomas Mührke