cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: Stalls when PUT-ing to Amazon S3

From: Dustin J. Mitchell <dustin_at_zmanda.com>
Date: Fri, 19 Dec 2008 14:01:44 -0500

On Thu, Dec 18, 2008 at 6:24 PM, koettermarkus_at_gmx.de
<koettermarkus_at_gmx.de> wrote:
> * amazons acks are coming in way to slow, at packet #1800 amazon
> acks=871627, #1801 sends packet seqnr=1737715
>
> * packet #4674, amazon sends an ack=3866967, acking packet #4482 - 2
> minutes later.

Yeah, Amazon's network implementation is extremely wacky. Sometimes
it uses SACK, sometimes it doesn't.. sometimes I could swear it's
sending pre-emptive ACKs, but sometimes the ACKs lag by a long time.

> * 192.168/16 made me think you are masqueraded and your router rewrites the
> packets, maybe mtu clamping renders the pcap useless?
>
> * captured on a direct connected system or the router instead?

No, no different. Nikolas's machine is NAT'd, but mine (where I see
the same things) is not.

> It is really interesting, but can't be curls fault.

You are correct. It turned out, as you might expect, to be a
combination of lots of several things. First, S3 fails a lot, in
weird ways -- they acknowledge this, and recommend retrying on
failure. Second, packet traces of S3 transactions will lead you down
lots of false alleys of weird TCP/IP behavior that do *not*
necessarily cause failures. Third, the changes to Amanda caused
retries to fail to "rewind" the buffer from which the request body was
being transmitted.

The result was that when S3 failed partway through a PUT, the second
request (retry) began partway through the buffer, and curl stopped
writing data when the READFUNC returned 0 at the end of the buffer.
This caused S3 to time out on this request, triggering a new retry.
That retry, and the subsequent 11 retries before giving up, did not
transmit any request body, which of course caused another timeout.

It was an interesting object lesson in debugging fault-tolerant systems.

Dustin

-- 
Storage Software Engineer
http://www.zmanda.com
Received on 2008-12-19