Hi,

During an application review I came to the issue of checking whether the
application handles all error conditions or not.

The program uploads a file to a remote HTTP server with a somewhat
semantically equivalent code (expressed here in python, the real code
does a multi upload):

filesize = os.path.getsize(filename)
c = pycurl.Curl()
c.setopt(pycurl.CONNECTTIMEOUT, 3)
#c.setopt(pycurl.TIMEOUT, 3)
c.setopt(pycurl.URL, url)
c.setopt(pycurl.UPLOAD, 1)
c.setopt(pycurl.READFUNCTION, open(filename, 'rb').read)
c.setopt(pycurl.INFILESIZE, filesize)
c.setopt(c.VERBOSE, 1)
try:
         c.perform()
except Exception:
         pass
print ("Uploaded %d" % c.getinfo(pycurl.SIZE_UPLOAD))
c.close()

This program has two problems:
1. if the TIMEOUT line is there, the uploadable file size is limited by
the upload bandwidth. It will time out in 3 seconds, no matter the data
is still flowing. On a gigabit NIC (let's say it has 100 MiBps BW) the
maximum file size will be 300 MiB (plus minus some rounding error ofc),
anything larger will die with timeout kicking in [1].
2. if the TIMEOUT line is not there, if the remote server hangs -but
keeps the TCP open, like when the kernel is alive, but the disk
subsystem dies in a way that the program stucks in disk wait state-, the
above program will never finish.

Of course the app could check SIZE_UPLOAD in regular intervals or
provide some progress function to do the same, but these increase
complexity pretty much.

Are there any other methods provided by curl, which I may have overlooked?

If not, what do you think about a new timeout setting, which detects
inactivity on the socket?
Setting for example INACTIVITY_TIMEOUT to 3 (seconds) would mean it will
fire the same way as current TIMEOUT, but only when there were no
activity on the socket (in either way, so it can be used for up- and
downloads).

Thanks,

[1]: timeout==3
* Trying 192.168.0.100...
* Connected to srv (192.168.0.100) port 8000 (#0)
> PUT /test HTTP/1.1
Host: srv:8000
User-Agent: PycURL/7.21.5 libcurl/7.47.0 OpenSSL/1.0.1q zlib/1.2.8
Accept: */*
Content-Length: 1073741824
Expect: 100-continue

< HTTP/1.1 100 Continue
* Operation timed out after 3006 milliseconds with 0 bytes received
* Closing connection 0
Uploaded 67509288

-------------------------------------------------------------------
List admin: https://cool.haxx.se/list/listinfo/curl-users
FAQ: https://curl.haxx.se/docs/faq.html
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2016-02-29