I am able to consistently reproduce this behavior with pipeline enabled.

*Test case:*

40 easy handles are used to send 2000 GET requests.

1. Set up 40 easy handles with *pipeline enabled and fresh connect = 1.*
2. Add all of them to multi interface. This results in 40 concurrent
connections.
3. In "CURLINFO_HEADER_OUT" in CURLOPT_DEBUGFUNCTION callback, check if the
handle has sent more than 50 requests, if so, set forbid reuse on this
handle.
4. In curl_multi_info_read, read the response, update the stats, remove
handle from multi, reset handle, set it again with* fresh connect = 0* and
add it back to the multi interface.
If this is the last transfer that this handle has to do, remove it from the
multi and clean up the easy handle.

*Observation:*
1. More than 40 connections are set up.
2. Wireshark captures 2000 200 OK responses but multi info read with curl
ok is called only around 1980+ times.
3. This is not reproducible if pipeline is disabled.

*Questions:*

1. What is the criteria for creating new connections?
 For example, initially 40 connections are created and subsequent transfers
should use the 40 connections. Instead, I am seeing around 5-10 additional
connections getting created. When does it create new connection? IF
pipeline is enabled, shouldn't it just pipeline onto an existing connection?

2. Shouldn't multi info read be called each time response from server, in
this case 200 OK is received?

3. IF 4 GET requests is pipelined, will CURLOPT_DEBUGFUNCTION callback will
be called 4 times or one time. I'm guessing it is one time. So how do we
account for all the requests sent in case it is pipelined? Is it possible
to know from an handle that it pipelined and has sent 4 GET requests?

On Tue, Oct 7, 2014 at 12:19 PM, nikhil ap <niks3089_at_gmail.com> wrote:

> "And by confirming you mean you saw the 200 response arrive locally from
> the network? "
> Yes I did.
>
> "And those 5 still "not processed" responses, they were really complete
> (ie had ending indicators according to the HTTP 1.1 spec) and ended or how
> do you mean "not processed" exactly? "
>
> That's right. I saw 200 OK responses captured in Wireshark for 200 GETs
> but I was handed over 195.
>
> "What did the counter in curl_multi_perform() say about number of pending
> transfers?"
> It didn't show any. I also added a static counter in multi_addmsg()
> function and printed the counter. It was 195.
>
> "What libcurl version on what platorm are you using?"
> Curl-7.37
>
> "Was this only plain HTTP requests?"
> Yes
>
> "Can you repeat the problem with (much) fewer requests? Like 5?"
> This is not seen with less number of requests.
>
> "After 2000K what? Are you talking about TCP reset? Can you please
> elaborate on what you mean."
> I meant 200 OK. Yes connection is getting TCP reset. This is less
> frequently seen.
>
> If it helps the code is modelled like the one in hiperfifo.c
>
> On Mon, Oct 6, 2014 at 5:31 PM, Daniel Stenberg <daniel_at_haxx.se> wrote:
>
>> On Mon, 6 Oct 2014, nikhil ap wrote:
>>
>> For example, 200 requests are sent but only 195 responses are processed
>>> even though 200 OK responses are received by the client.
>>> I used wireshark to confirm that all requests are getting responses.
>>>
>>
>> And by confirming you mean you saw the 200 response arrive locally from
>> the network? And those 5 still "not processed" responses, they were really
>> complete (ie had ending indicators according to the HTTP 1.1 spec) and
>> ended or how do you mean "not processed" exactly?
>>
>> What did the counter in curl_multi_perform() say about number of pending
>> transfers?
>>
>> What libcurl version on what platorm are you using? Was this only plain
>> HTTP requests? Can you repeat the problem with (much) fewer requests? Like
>> 5?
>>
>> Which means curl_multi_info_read() is handing over only 195 responses.
>>> This is not a right behavior. Any idea as to why this could happen?
>>>
>>
>> I'm not convinced your analysis is correct so I cannot really comment on
>> exactly what you see and why. I've never seen this happen nor heard it
>> reported before and right now I can't think of a way it can happen. Other
>> than that they in fact maybe didn't really end and thus libcurl was waiting
>> for the remainder of the reponses...
>>
>> In certain other cases, after 200OK is processed but curl is immediately
>>> sending reset. I would also like to know why.
>>>
>>
>> After 2000K what? Are you talking about TCP reset? Can you please
>> elaborate on what you mean.
>>
>> Is there a way to see what happens behind the scene? Like enable logging
>>> so
>>> that I can see info level and error logs generated by libcurl
>>>
>>
>> We offer CURLOPT_VERBOSE and CURLOPT_DEBUGFUNCTION, but no more info than
>> so. The next step after that is plain old debugging and added
>> infof()/printf() statements!
>>
>> --
>>
>> / daniel.haxx.se
>> -------------------------------------------------------------------
>> List admin: http://cool.haxx.se/list/listinfo/curl-library
>> Etiquette: http://curl.haxx.se/mail/etiquette.html
>
>
>
>
> --
> niks
>

-- 
niks

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2014-10-08