On Tue, 24 Jul 2012, George Rizkalla wrote:

> We are currently looking at contributing to libcurl's pipelining
> implementation, and we were hoping to get your feedback on some areas we'd
> like to help with.

Awesome. You are most welcome here and I'm sure I'm not the only one who will
appreciate this!

Let me give you a word of advice already now: work on making sure we can
exercise most of the pipelining code and features in the test suite.
Pipelining is tricky already by default, if we can't verify functionality and
track down problems with the test suite things often tend to fade and break
while we're not paying attention. I believe the existing pipelining support
suffers a bit from this.

> 1. MAX SOCKETS CHANGES

> While CURLMOPT_MAXCONNECTS imposes a limit on the number of sockets
> persisted, it does not regulate the maximum number of sockets that are in
> use at a given point in time. It is proposed that CURLMOPT_MAXCONNECTS be
> aliased/renamed CURLMOPT_MAXCONNECTS_SOFT, and that a new option,
> CURLMOPT_MAXCONNECTS_HARD would be introduced. The latter option would
> regulate the maximum number of open sockets at any given point in time. This
> will be necessary to eliminate the need for queuing requests at the
> application layer when the application wishes to throttle the number of
> underlying sockets.

I know this has been requested by others in the past - exactly to prevent the
application from having to do the queing. I suspect you want to do this by not
altering the API and simply just not start new handles if there aren't any
available connections ? I figure that could be a fairly simple change.

> 2. HTTP PIPELINING CHANGES
> -Max Pipeline Length
> It is proposed that the maximum pipeline length is set using a new curl
> option, CURLMOPT_MAX_PIPELINE_LENGTH. When the number of outstanding
> requests exceeds this parameter, a new socket should be opened. When the
> number of outstanding requests reaches CURLMOPT_MAX_PIPELINE_LENGTH *
> CURLMOPT_MAX_CONNECTS_HARD, requests are queued.

Makes perfect sense.

> -Socket Penalization Principle:
> For effective pipelining, the concept of socket penalization should be
> introduced to libcurl. The guiding tenet is that, where possible, a new
> request should not be pipelined behind a request (or response) that is
> known to be large.
>
> It is possible to determine whether a socket should be penalized by either:
>
> a) A Content-Length header specifying a content-length greater than the
> proposed new curl option CURLMOPT_CONTENT_LENGTH_PENALTY_SIZE
> b) A chunk size larger than the proposed new curl option
> CURLMOPT_CHUNK_PENALTY_SIZE where a chunked transfer-encoding is used
>
> Of course, if a chunked transfer-encoding is used with multiple chunks
> smaller than CURLMOPT_CHUNK_PENALTY_SIZE, it will not be possible to
> identify the penalized condition correctly.

This sounds fine too. I'm a bit doubtful about the ncesssity for the
CURLMOPT_*_PENALTY_SIZE options though. They seem like options that simply
nobody would touch if we would claim the defaults are somewhat sensible. Or do
you see any use-case in your world where you will actually change these
according to some scheme or acquired knowledge in the application?

> -Pipeline-able Sockets

> In addition to the obvious checks for the use of an HTTP/1.1 server, before
> assuming that pipelining is possible on a particular socket, libcurl could
> check for older/incompatible IIS servers before deciding to pipeline.

Ouch. How would that be done? Scanning for certain magic keywords in response
Server: headers?

> If the first response to a request on a socket is marked as HTTP/1.0, or an
> older IIS server version is used, or the site is black-listed (see below),
> the socket should be characterized as CAN_PIPELINE = false.

This makes the blacklist basically always just grow. Will there be any means
of purging old entries or don't you see this as a problem?

> -Basic Socket Selection Algorithm
[snip]

Wow. Sounds like you've been giving this some thoughts and I am impressed. It
certainly sounds like a working way that will *greatly* enhance the way
libcurl does pipelining.

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html