On Wed, 8 Oct 2014, nikhil ap wrote:

> I am able to consistently reproduce this behavior with pipeline enabled.

Then please consider writing up an as small example as possible that repeats
the problem and post it for others to try out and possibly analyze.

Pipelining is tricky and an underused feature in libcurl so I'm sure there are
problems left in there to fix with it. Since it is also a bit timing sensitive
and depending on server side behavior, it gets hairy at times.

> 40 easy handles are used to send 2000 GET requests.
>
> 1. Set up 40 easy handles with *pipeline enabled and fresh connect = 1.*
> 2. Add all of them to multi interface. This results in 40 concurrent
> connections.
> 3. In "CURLINFO_HEADER_OUT" in CURLOPT_DEBUGFUNCTION callback, check if the
> handle has sent more than 50 requests, if so, set forbid reuse on this
> handle.
> 4. In curl_multi_info_read, read the response, update the stats, remove
> handle from multi, reset handle, set it again with* fresh connect = 0* and
> add it back to the multi interface.
>
> If this is the last transfer that this handle has to do, remove it from the
> multi and clean up the easy handle.

Just don't do the mistake and assume that the easy handle is associated with a
connection after you've removed it from the multi handle. An easy handle is
associated with a connection (and transfer) while connected to a multi handle,
but not anymore when removed from it. I'm not saying you do, I'm just trying
to clarify.

> 1. More than 40 connections are set up.

The description above doesn't limit the number of connections to 40 though -
it'll keep old connections around in the connection cache for example. Are you
communicating with more than 40 hosts?

> 1. What is the criteria for creating new connections? For example, initially
> 40 connections are created and subsequent transfers should use the 40
> connections. Instead, I am seeing around 5-10 additional connections getting
> created.

You might want CURLMOPT_MAX_HOST_CONNECTIONS.

> When does it create new connection? IF pipeline is enabled, shouldn't it
> just pipeline onto an existing connection?

Doesn't it? What does the verbose log tell you?

Using a single connection is often inferior even with pipelining, so libcurl
allows you to set CURLMOPT_MAX_HOST_CONNECTIONS and have the requests
distributed over N connections.

There's also a maximum pipe length (CURLMOPT_MAX_PIPELINE_LENGTH) which limits
how many requests libcurl will put in a single pipe before it creates another
connection (or queues up the request).

> 2. Shouldn't multi info read be called each time response from server, in
> this case 200 OK is received?

You can basically call it whenever you want, but it should return CURLMSG_DONE
once for every transfer that completes (successful or not).

It does make sense to call curl_multi_info_read() immediately after calls to
curl_multi_socket_action() or curl_multi_perform().

> 3. IF 4 GET requests is pipelined, will CURLOPT_DEBUGFUNCTION callback will
> be called 4 times or one time.

CURLOPT_DEBUGFUNCTION will or can be called _many_ times for each request as
far as I remember. We don't really give any promises or guarantees other than
what is already documented.

> I'm guessing it is one time.

No. It is probably most often once per request but there can indeed be more.
It depends.

> So how do we account for all the requests sent in case it is pipelined? Is
> it possible to know from an handle that it pipelined and has sent 4 GET
> requests?

I don't think we expose that info in any particular way through the API today.
We probably would need to add a counter somewhere and offer a way to export
that counter.

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html