On Wed, 8 Feb 2012, Felix E. Klee wrote:

> What can be seen here:
>
> 1. A POST request is issued.
> 2. No communication for 40s. (= CURLOPT_TIMEOUT, CURLOPT_CONNECTTIMEOUT)
> 3. There is brief communication. I assume that cURL tells the server to
> give up (FIN, ACK), and the server responds quickly (ACK).

This is when libcurl closes the connection, as you also saw that libcurl
legitimately timed out without the server sending anything.

Is the POST data acked by the server?

> 3. No communication for 159s.
> 3. The server responds with RST, ACK.

I haven't read the spec lately but it seems odd that it would keep the
connection alive for 160 seconds after it acked the FIN and then send a RST on
it. But maybe I'm just not remembering my TCP details good enough.

> So far I saw the problem only twice today, and both times it was the
> same pattern:
>
> 1. query: no problem
>
> 2. long pause of 30 to 60 min
>
> 3. another query: timeout hit

So how long pause do you need to make the problem trigger? I would guess that
you see this problem because something silently kills/discards the connection
after N minutes so that if you're just faster than N it'll run fine. You can
possibly overcome this problem by using TCP keep-alive on the connection to
avoid the whatever-it-is-that-kills-the-connection to stop doing it.

That "whatever" can of course be a firewall or load balancer or similar on the
server side.

> Background information: This is web-scraping that I am implementing for a
> customer. While my customer is in contact with the airline company that runs
> the servers, I doubt that they will change anything especially for us.
> According to my customer, they do not even provide an API (which would make
> things much more straight forward). So, there is no choice other than to
> work around the problem client-side.

I'm not saying otherwise - the real world is a rough place and not at all kind
to us who just want to adhere to protocol specs and be fine with it.

I'm just saying that this isn't a problem in libcurl itself, or at least we
haven't seen any evidence of a bug or flaw in libcurl that causes this.

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html