cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: Curl retry without notifying application?

From: Mark Aldred <mark_at_twinstrata.com>
Date: Thu, 27 Aug 2009 10:56:28 -0400

On Thu, Aug 27, 2009 at 10:44 AM, Mark Aldred <mark_at_twinstrata.com> wrote:

>
>
>>
>>
>> I am using libcurl 7.19.6.
>>>
>>> My application uses CURLOPT_READFUNCTION and CURLOPT_WRITEFUNCTION copy
>>> the data into and out of curl. My callback functions us structures to track
>>> how much data has been copied to maintain sane pointers as the copy
>>> progresses. Fairly standard stuff. When curl return an error to our
>>> application my code resets the pointers used in the data copy and retires
>>> the curl request (for retryable errors). All this works fine.
>>>
>>> Today I encountered a situation where it appears I issued a PUT request
>>> to curl. Curl got the 100-continue from the host, invoked by copy function
>>> to get all the data then encountered an error. It seems that curl then
>>> retried the operation without returning an error to my application so that I
>>> can reset the copy pointers in my structures. The result is that when curl
>>> invoked the copy function in my application for the retry, my copy function
>>> returned zero because it thought (correctly) that all the data had been
>>> copied already. At this point the request hangs.
>>>
>>> Does curl retry requests without returning an error to the application so
>>> that it can prepare for the retry?
>>>
>>> Here is an excerpt from my log.
>>>
>>> 0x14f7ae0 2009-Aug-26 23:22:11.905625 CURLINFO_HEADER_OUT 23006848 PUT
>>> /some_url HTTP/1.1
>>> Accept: */*
>>> Authorization: AWS 1VVN4RW1K6ACA2246MR2:jONA2hLltDtR7WSR84/L0xsGH8g=
>>> content-length: 131072
>>> content-type: application/octet-stream
>>> date: Thu, 27 Aug 2009 03:22:11 GMT
>>> host: s3.amazonaws.com
>>> Expect: 100-continue
>>>
>>> 0x14f7ae0 2009-Aug-26 23:22:12.912369 CURLINFO_TEXT 23006848 Done waiting
>>> for 100-continue
>>> 0x14f7ae0 2009-Aug-26 23:22:12.912508 _OutboundDataCopy COPY eh:23006848
>>> copy_size:40960 iovec size:131072 copy size:40960 remaining:131072
>>> 0x14f7ae0 2009-Aug-26 23:22:12.913001 _OutboundDataCopy COPY eh:23006848
>>> copy_size:40960 iovec size:131072 copy size:40960 remaining:90112
>>> 0x14f7ae0 2009-Aug-26 23:22:12.913454 _OutboundDataCopy COPY eh:23006848
>>> copy_size:40960 iovec size:131072 copy size:40960 remaining:49152
>>> 0x14f7ae0 2009-Aug-26 23:22:12.913904 _OutboundDataCopy COPY eh:23006848
>>> copy_size:8192 iovec size:131072 copy size:8192 remaining:8192
>>> 0x14f7ae0 2009-Aug-26 23:22:12.914067 _OutboundDataCopy DONE bytes
>>> processed:131072 size:131072 eh:23006848
>>>
>>> 0x14f7ae0 2009-Aug-26 23:22:24.665157 CURLINFO_TEXT 23006848 SSL read:
>>> error:00000000:lib(0):func(0):reason(0), errno 104
>>> 0x14f7ae0 2009-Aug-26 23:22:24.665243 CURLINFO_TEXT 23006848 Connection
>>> died, retrying a fresh connect
>>> 0x14f7ae0 2009-Aug-26 23:22:24.665283 CURLINFO_TEXT 23006848 Closing
>>> connection #0
>>> 0x14f7ae0 2009-Aug-26 23:22:24.668640 CURLINFO_TEXT 23006848 Issue
>>> another request to this URL: 'https://s3.amazonaws.com:443/some_url<https://s3.amazonaws.com/some_url>'
>>>
>>> 0x14f7ae0 2009-Aug-26 23:22:24.668719 CURLINFO_TEXT 23006848 Re-using
>>> existing connection! (#1) with host s3.amazonaws.com
>>> 0x14f7ae0 2009-Aug-26 23:22:24.668757 CURLINFO_TEXT 23006848 Connected to
>>> s3.amazonaws.com (72.21.202.97) port 443 (#1)
>>>
>>> 0x14f7ae0 2009-Aug-26 23:22:24.741157 CURLINFO_HEADER_IN 23006848
>>> HTTP/1.1 100 Continue
>>> 0x14f7ae0 2009-Aug-26 23:22:24.753208 _OutboundDataCopy DONE bytes
>>> processed:131072 size:131072 eh:23006848
>>>
>>> One last thing. What is this SSL read error?
>>>
>>> Thank you for any help you can provide.
>>>
>>>
>> This comment in Curl_retry_request() indicates that curl is doing exactly
> what I thought. Sounds like it is retrying a PUT without returning an
> error. This behavior is fine for GETs and DELETEs, but not PUTs.
>
> conn->bits.retry = TRUE; /* mark this as a connection we're about
>
>>
>> to retry. Marking it this way should
>> prevent i.e HTTP transfers to return
>> error just because nothing has been
>> transfered! */
>>
>>
>> I'm trying this fix in Curl_http_done(). does this make sense?
>>
>> if(!premature && /* this check is pointless when DONE is called before
>> the
>> entire operation is complete */
>> !conn->bits.retry &&
>> data->req.upload_present != 0 && /////////////////////////
>> ADDED THIS LINE
>> ((http->readbytecount +
>> data->req.headerbytecount -
>> data->req.deductheadercount)) <= 0) {
>> /* If this connection isn't simply closed to be retried, AND nothing
>> was
>> read from the HTTP server (that counts), this can't be right so we
>> return an error here */
>> failf(data, "Empty reply from server");
>> return CURLE_GOT_NOTHING;
>> }
>>
>
> Sorry for the Top Post
>
> --
> Mark
>
>
This fix appears to be working for me. I see the "SSL read: error" on a put
request and then I see this error being returned back to my application so
that I can prepare for the retry. I would appreciate a comment on the
correctness of this fix. Thanks.

-- 
Mark
Received on 2009-08-27