On Tue, Jul 24, 2012 at 03:01:26PM +0000, George Rizkalla wrote:
> The proposed algorithm involves balancing HTTP requests over multiple TCP
> sockets, while avoiding use of HTTP pipelining in instances where we
> believe errors are likely to occur, or where it is likely that there would
> be a performance hit if pipelining is used.

This is set of proposals sounds really useful, but I wonder if it's
better kept in the application rather than in libcurl. Most of the
suggestions can be done today without changing (and complicating)
libcurl, using its existing interfaces and callbacks. And some of them
only make sense with additional data that only the application can
provide, which would require numerous additional options and interfaces
to bring into libcurl.

Really what this calls for is a layer sitting above libcurl that takes
care of queuing handles into appropriate pipelines, creating new ones
as necessary and optimizing the balance of requests to sockets as
appropriate. The low-level requests can already be performed by
libcurl--this would really be a "value added" layer that would only be
used by applications that need it.

I guess what I've had in the back of my mind for a while is a new
library, let's call it "libcurlapp", sitting on top of libcurl. This
could be a place for those numerous suggestions over the years for
features that don't quite make sense in a low-level library like
libcurl, but would still be really useful for some applications.
Things like sophisticated proxy selection, memory and disk caching
of results, handle pools, Metalink support, and now pipeline
optimization. Those sort of features would IMHO be well-suited for a
libcurlapp, even if not in libcurl itself.

I'm not volunteering to lead such a project :-) or trying to dampen the
enthusiasm for improved pipelining support in libcurl, just throwing
the idea out there to consider. I could see how such a feature set
would be useful, but I'm not sure whether it could be done in such a way
that's general enough for use a wide-range of different applications.
But, if it's written to use stock libcurl underneath, then it ought to
be able to be designed in such a way that apps could mix and match just
those additional features they want out of the new library, while
leaving the full power of libcurl available as well.

>>> Dan
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2012-07-25