cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: Curl retry without notifying application?

From: Mark Aldred <mark_at_twinstrata.com>
Date: Thu, 27 Aug 2009 11:01:14 -0400

On Thu, Aug 27, 2009 at 10:56 AM, Mark Aldred <mark_at_twinstrata.com> wrote:

>
>
> On Thu, Aug 27, 2009 at 10:44 AM, Mark Aldred <mark_at_twinstrata.com> wrote:
>
>>
>>
>>>
>>>
>>> I am using libcurl 7.19.6.
>>>>
>>>> My application uses CURLOPT_READFUNCTION and CURLOPT_WRITEFUNCTION copy
>>>> the data into and out of curl. My callback functions us structures to track
>>>> how much data has been copied to maintain sane pointers as the copy
>>>> progresses. Fairly standard stuff. When curl return an error to our
>>>> application my code resets the pointers used in the data copy and retires
>>>> the curl request (for retryable errors). All this works fine.
>>>>
>>>> Today I encountered a situation where it appears I issued a PUT request
>>>> to curl. Curl got the 100-continue from the host, invoked by copy function
>>>> to get all the data then encountered an error. It seems that curl then
>>>> retried the operation without returning an error to my application so that I
>>>> can reset the copy pointers in my structures. The result is that when curl
>>>> invoked the copy function in my application for the retry, my copy function
>>>> returned zero because it thought (correctly) that all the data had been
>>>> copied already. At this point the request hangs.
>>>>
>>>> Does curl retry requests without returning an error to the application
>>>> so that it can prepare for the retry?
>>>>
>>>> Here is an excerpt from my log.
>>>>
>>>> 0x14f7ae0 2009-Aug-26 23:22:11.905625 CURLINFO_HEADER_OUT 23006848 PUT
>>>> /some_url HTTP/1.1
>>>> Accept: */*
>>>> Authorization: AWS 1VVN4RW1K6ACA2246MR2:jONA2hLltDtR7WSR84/L0xsGH8g=
>>>> content-length: 131072
>>>> content-type: application/octet-stream
>>>> date: Thu, 27 Aug 2009 03:22:11 GMT
>>>> host: s3.amazonaws.com
>>>> Expect: 100-continue
>>>>
>>>> 0x14f7ae0 2009-Aug-26 23:22:12.912369 CURLINFO_TEXT 23006848 Done
>>>> waiting for 100-continue
>>>> 0x14f7ae0 2009-Aug-26 23:22:12.912508 _OutboundDataCopy COPY eh:23006848
>>>> copy_size:40960 iovec size:131072 copy size:40960 remaining:131072
>>>> 0x14f7ae0 2009-Aug-26 23:22:12.913001 _OutboundDataCopy COPY eh:23006848
>>>> copy_size:40960 iovec size:131072 copy size:40960 remaining:90112
>>>> 0x14f7ae0 2009-Aug-26 23:22:12.913454 _OutboundDataCopy COPY eh:23006848
>>>> copy_size:40960 iovec size:131072 copy size:40960 remaining:49152
>>>> 0x14f7ae0 2009-Aug-26 23:22:12.913904 _OutboundDataCopy COPY eh:23006848
>>>> copy_size:8192 iovec size:131072 copy size:8192 remaining:8192
>>>> 0x14f7ae0 2009-Aug-26 23:22:12.914067 _OutboundDataCopy DONE bytes
>>>> processed:131072 size:131072 eh:23006848
>>>>
>>>> 0x14f7ae0 2009-Aug-26 23:22:24.665157 CURLINFO_TEXT 23006848 SSL read:
>>>> error:00000000:lib(0):func(0):reason(0), errno 104
>>>> 0x14f7ae0 2009-Aug-26 23:22:24.665243 CURLINFO_TEXT 23006848 Connection
>>>> died, retrying a fresh connect
>>>> 0x14f7ae0 2009-Aug-26 23:22:24.665283 CURLINFO_TEXT 23006848 Closing
>>>> connection #0
>>>> 0x14f7ae0 2009-Aug-26 23:22:24.668640 CURLINFO_TEXT 23006848 Issue
>>>> another request to this URL: 'https://s3.amazonaws.com:443/some_url<https://s3.amazonaws.com/some_url>'
>>>>
>>>> 0x14f7ae0 2009-Aug-26 23:22:24.668719 CURLINFO_TEXT 23006848 Re-using
>>>> existing connection! (#1) with host s3.amazonaws.com
>>>> 0x14f7ae0 2009-Aug-26 23:22:24.668757 CURLINFO_TEXT 23006848 Connected
>>>> to s3.amazonaws.com (72.21.202.97) port 443 (#1)
>>>>
>>>> 0x14f7ae0 2009-Aug-26 23:22:24.741157 CURLINFO_HEADER_IN 23006848
>>>> HTTP/1.1 100 Continue
>>>> 0x14f7ae0 2009-Aug-26 23:22:24.753208 _OutboundDataCopy DONE bytes
>>>> processed:131072 size:131072 eh:23006848
>>>>
>>>> One last thing. What is this SSL read error?
>>>>
>>>> Thank you for any help you can provide.
>>>>
>>>>
>>> This comment in Curl_retry_request() indicates that curl is doing exactly
>> what I thought. Sounds like it is retrying a PUT without returning an
>> error. This behavior is fine for GETs and DELETEs, but not PUTs.
>>
>> conn->bits.retry = TRUE; /* mark this as a connection we're about
>>
>>>
>>> to retry. Marking it this way should
>>> prevent i.e HTTP transfers to return
>>> error just because nothing has been
>>> transfered! */
>>>
>>>
>>> I'm trying this fix in Curl_http_done(). does this make sense?
>>>
>>> if(!premature && /* this check is pointless when DONE is called before
>>> the
>>> entire operation is complete */
>>> !conn->bits.retry &&
>>> data->req.upload_present != 0 && /////////////////////////
>>> ADDED THIS LINE
>>> ((http->readbytecount +
>>> data->req.headerbytecount -
>>> data->req.deductheadercount)) <= 0) {
>>> /* If this connection isn't simply closed to be retried, AND nothing
>>> was
>>> read from the HTTP server (that counts), this can't be right so we
>>> return an error here */
>>> failf(data, "Empty reply from server");
>>> return CURLE_GOT_NOTHING;
>>> }
>>>
>>
>> Sorry for the Top Post
>>
>> --
>> Mark
>>
>>
> This fix appears to be working for me. I see the "SSL read: error" on a
> put request and then I see this error being returned back to my application
> so that I can prepare for the retry. I would appreciate a comment on the
> correctness of this fix. Thanks.
>
>
> --
> Mark
>
>
Sorry, I spoke too soon. I got two of the SSL read: erorrs. My attempted
fix appeared to get the correct behavior on the first on but not on the
second one. Just a reminder that at this point I'm not trying to fix the
SSL read errors. I want to make sure that they are handled correctly.
Thoughts appreciated...

-- 
Mark
Received on 2009-08-27