> From: curl-library-bounces_at_cool.haxx.se [curl-library-bounces_at_cool.haxx.se] on
> behalf of Dan Fandrich [dan_at_coneharvesters.com]
> Sent: Wednesday, July 25, 2012 5:30 PM
> To: curl-library_at_cool.haxx.se
> Subject: Re: HTTP Pipelining Contributions
>
> This is set of proposals sounds really useful, but I wonder if it's
> better kept in the application rather than in libcurl. Most of the
> suggestions can be done today without changing (and complicating)
> libcurl, using its existing interfaces and callbacks. And some of them
> only make sense with additional data that only the application can
> provide, which would require numerous additional options and interfaces
> to bring into libcurl.

Can they? curl today doesn't expose the number of connections and the mapping of curl handles to connections. The only options I can find to control how a request is mapped to a connection are
The only callbacks I can find today that can be used to control how requests are assigned to connections are CURLOPT_FRESH_CONNECT and CURLOPT_FORBID_REUSE, which force requests to use new connections (and disable both pipelining and regular connection reuse).

I don't see any way to implement this proposal outside curl without adding functions to assign requests to connections explicitly, and I thought that Daniel was strongly against that.

> Really what this calls for is a layer sitting above libcurl that takes
> care of queuing handles into appropriate pipelines, creating new ones
> as necessary and optimizing the balance of requests to sockets as
> appropriate. The low-level requests can already be performed by
> libcurl--this would really be a "value added" layer that would only be
> used by applications that need it.

What would the interface to curl look like? curl owns the concepts of connections and requests, and in fact generates some requests (such as HTTP auth followups or CONNECTs) internally. So it would need to call into the pipelining library for each request asking what to do with it. And when it wanted to open a socket, it would need to call into the pipelining library for permission (if the pipelining library does load balancing, it's in charge of enforcing connection limits) - and if it returned false, do what? When it's time to close a socket, curl would need to call into the pipelining library to decide which one to get rid of. That's a lot of callbacks, and curl would need to have a default path for people not using the pipelining library as well.

The hardest part of this proposal is adding a request queue to curl, and I think that would be needed even with a pipelining library, since if we add any limits on the number of simultaneous requests, curl is going to have to put them somewhere.

Unless the idea is that the client code interfaces with the pipelining library instead of with curl directly, and the pipelining library queues requests and doesn't send them to curl until it's decided a connection exists to handle them. But this means a design where the pipelining library drives curl, which is sort of an inversion of the interface that curl uses today. It would mean that it's hard to add support for this library to an app that's already using curl, and probably a lot of redesign of curl's interface itself (for example, if the pipelining library is driving, that conflicts with curl's automatic generation of auth followups and CONNECTs). Sounds like a can of worms.

> I guess what I've had in the back of my mind for a while is a new
> library, let's call it "libcurlapp", sitting on top of libcurl. This
> could be a place for those numerous suggestions over the years for
> features that don't quite make sense in a low-level library like
> libcurl, but would still be really useful for some applications.
> Things like sophisticated proxy selection, memory and disk caching
> of results, handle pools, Metalink support, and now pipeline
> optimization. Those sort of features would IMHO be well-suited for a
> libcurlapp, even if not in libcurl itself.

I think this sounds like a great idea, since the internals of curl are pretty hairy and so separating concerns into another layer with well-defined boundaries would help with complexity a lot. But some things which would logically be part of this are already in curl, so we'd want to refactor it - but if we move functionality out of curl into this new library, how do we maintain compatibility? I wouldn't know where to begin to start with this.

Joe
---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2012-07-26