Re: Failed sending data to the peer in 7.72.0

I went through commits on master that aren't in 7.72.0 and came across
728f8d3bd
<https://github.com/curl/curl/commit/728f8d3bdc336e3fa838f45cad6c0133a6b604ae>,
which looks promising. It says it partially reverts a change from 7.65.2,
which is kind of weird because this problem does *not *occur for me in
7.71.1, which suggests that one of the other changes in 7.72.0 (maybe the
schannel stuff?) somehow had some spooky-action-at-a-distance and made this
much worse in 7.72.0. I feel like 728f8d3bd still has a race condition if
the FIN has been issued by the server but not yet processed by poll, but
that's neither here nor there.

Anyhow, I cherry-picked 728f8d3bd and it does seem to have fixed my issue
(or, at least, my issue hasn't occurred in the intervening ~30 minutes).

On Mon, Sep 14, 2020 at 2:38 PM James Brown <jbrown_at_easypost.com> wrote:

> Unfortunately, it takes rather several hours of running to identify this
> failure; building a new package and bisecting is going to be quite
> difficult with that long of a turnaround.
>
> I however was able to get a tcpdump of one of the affected sessions.
>
> This session spans several requests to the same backend (Typhoeus pools
> curl sockets and tries to use keep-alive whenever possible). The first few
> requests succeed; at some point there's a ~9-second break in requests while
> the process does other stuff and the server closes the TCP stream (our load
> balancer is configured to close idle keep-alive sessions after 9 seconds).
> Within a few milliseconds, another request is sent on the same socket; it
> immediately gets back a RST because the server has already closed this
> socket. That appears to cause this exception to be raised by libcurl.
>
> Did anything change around handling keep-alive session expiry in HTTP/1.1
> mode? Nothing jumps out at me in the git log (there's something in the
> schannel backend, but I'm on Linux; there's also d5bb459ccf
> <https://github.com/curl/curl/commit/d5bb459ccf1fc5980ae4b95c05b4ecf6454a7599> which
> claims to only affect CONNECT-only connections, and all of this is regular
> GETs and POSTs)...
>
> On Fri, Sep 11, 2020 at 10:34 PM Ray Satiro via curl-library <
> curl-library_at_cool.haxx.se> wrote:
>
>> On 9/11/2020 2:03 PM, James Brown via curl-library wrote:
>>
>> After upgrading a test cluster from 7.71.1 to 7.72.0, we're now seeing
>> around 0.1% of POSTs from one (and only one) of our applications fail with
>> "Failed sending data to the peer" (CURLE_SEND_ERROR) and no other error.
>> Based on logs, the request actually succeeds, but libcurl is returning this
>> error. This application is using the Ruby Typhoeus wrapper and is itself
>> unchanged. The relevant connections are all HTTP/1.1 connections to hosts
>> on the local network, and the POSTs are all very small (<1KB) with nothing
>> interesting about them.
>>
>> I haven't had any luck tracking this down since it's such a low fraction
>> of requests and is only affecting one of our several hundred applications,
>> but it reproducibly happens with 7.72 and not with 7.71.1.
>>
>> Anyone have any suggestions for how to try to track down the regression?
>> I looked at the diff between 7.71.1 and 7.72.0 and no lines containing the
>> string "CURLE_SEND_ERROR" were touched, which is unfortunate.
>>
>>
>> There are no similar reports and I looked through the commit history but
>> nothing stood out. If you can reliably reproduce then try bisecting it
>> https://github.com/curl/curl/wiki/how-to-git-bisect
>>
>>
>> -------------------------------------------------------------------
>> Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
>> Etiquette: https://curl.haxx.se/mail/etiquette.html
>
>
>
> --
> James Brown
> Engineer
>

-- 
James Brown
Engineer

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2020-09-15