We've been encountering some strange stall behavior, where (lib)curl
seems to simply stop sending data. The errors are reproducible, but
sporadic. We're happy to try debugging suggestions as well as possible
solutions.

Overview:
1) Client (Amanda) sends a PUT request to server (Amazon S3)
2) Server responds with 100 Continue
3) Client sends some data
4) Client suddenly stops sending data
5) Server sends 400 Bad Request response with the (S3-specific)
RequestTimeout

Here's an example generated using the 20081217 snapshot:

Packets:
No. | rel. time | description
1558 | 2.910009 | data acknowledged by...
3083 | 5.103013 | ack acknowledged by...
3208 | 5.943238 | last packet sent by client before error response
3597 | 14.827490 | last packet (sent by server) before error response, acks
3598 | 34.892258 | error response

In the strace:
time | description
1229538079.157424 | next-to-last uploading print
1229538079.157616 | last call to send
1229538079.197150 | last successful poll(out) calls
1229538079.197463 | last uploading print
1229538079.197957 | first-in-batch poll timeout before error response
1229538109.232189 | last-in-batch poll timeout before error response
1229538109.232672 | successful poll(in) calls
1229538109.324232 | error response recv-ed

We first noticed this with libcurl 7.18.2, but have reproduced it with
7.19.2 and the 20081217 CVS snapshot. 7.18.12 behaves a bit differently
from the other two.

All libraries are unpatched and simply configured with --prefix, so the
following information is the same (except the libcurl version, of
course):
$ curl --version
curl 7.19.3-20081217 (i686-pc-linux-gnu) libcurl/7.19.3-20081217
OpenSSL/0.9.8g zlib/1.2.3 libidn/0.6.14
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: IDN IPv6 Largefile NTLM SSL libz

Excerpts of packet captures and straces from versions 7.18.2, 7.19.3,
and 20081217 snapshot:
http://nikolasco.s3.amazonaws.com/broken_s3.zip

Relevant code:
http://github.com/nikolasco/amanda/tree/trunk/device-src/s3.c#L954
http://github.com/nikolasco/amanda/tree/trunk/device-src/s3.c#L519
http://github.com/nikolasco/amanda/tree/trunk/device-src/s3.c#L745
(uploading debug print was added to s3_buffer_read_func)

Thanks,

-- 
        Nikolas Coukouma
        Design Engineer, Zmanda, Inc.
        http://www.zmanda.com/