Hi

We are using pycurl (more specifically, pycurl.CurlMulti() with
pycurl.Curl() as handle) for uploading large files with FTP. While it works
really nicely on most conditions, there are some picky firewalls which seem
to disconnect the control port, thus killing the transfers, resulting in an
"Timeout Reached" error on the pycurl-based client. The ftp server does not
support resuming, so resuming it after each disconnection isn't an option.

We managed to implement tcp-level keep-alive on the 2 sockets (control &
data, by using M_SOCKETFUNCTION, see code below), which AFAIK is the closest
to the approach taken by the original curl utility. We also raised timeouts
to 5 min for TCP and 24 hours for general timeout value.

However, this does not solve all of the disconnections issues, so we are
wanting to make it more robust by using NOOP-based keep alive (or LST,
etc...), i.e. not sending dummy tcp packets but rather dummy FTP commands.

Any idea how this could be done in python ? Do we need to patch libcurl
itself for this ? How can we continue interact with the command stream while
uploading ?

Thanks for any suggestion,

Florent

Context:
======

libcurl & pycurl versions: 7.18.2
Linux kernel: 2.6.27-14-generic

Code for tcp keep alive:
==================

def _socket_cb(self, event, socket_fd, user_data, socket_data):
    sock = socket.fromfd(socket_fd, socket.AF_INET, socket.SOCK_STREAM)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 60)
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 60)
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 20)

curl_multi.setopt(pycurl.M_SOCKETFUNCTION, self._socket_cb)

Raising timeouts options:
====================

(pycurl.FTP_RESPONSE_TIMEOUT, 300), # 5 min
(pycurl.TIMEOUT, 24 * 3600) # 24 hours

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2010-11-18