On Fri, 30 Dec 2011, Mukul Kedare wrote:

> It seems that above assumption was wrong, since we are using multi curl per
> thread and with multi curl, caching will happen at multi curl handle and not
> at easy handles. So for 100 threads, even though we restrict the number of
> curl easy handles for "URL_1" to 10, each with MAXCONNECT=1, we are seeing
> open connections >= 10 and <= 100. I assume this is due to use of multi
> handle at thread level. "URL_1" connections are getting cached at each
> thread (number of threads = 100).

Exactly!

> Q1. We want to make persistent open connections and restrict the number of
> open connections with URLs. Can you suggest the correct way in which we
> should be using curl multi and easy interfaces to solve this problem?

I can't think of any immediate easy fix for your problem. I understand what
you're trying to get out from libcurl, but its caching doesn't quite offer
that level of flexibilty that you'd like here.

You can possibly change your logic. Like if you would be more careful to only
add URL_1 to multi handles that should have it, it would limit the amount to
10. Or use some kind of counter of your own and use the OPENSOCKETFUNCTION to
fail socket creations when the counter for a specific address has reached 10
etc.

> Q2. Can we have a common curl connection cache shared between curl multi
> handles?

Not currently, no. But:

> Q3. Also I have read a thread in the mailing list (
> http://curl.haxx.se/mail/lib-2011-05/0017.html) where you stated that "Share
> Interface" can be used for sharing connection cache. Is this possible? :)

The share interface is our generic way to share certain kinds of data between
handles. It would make perfect sense to make sure a share object also can hold
the connection pool as then you would be able to make all 100 multi handles
use the *same* connection cache/pool and a single maximum limit would be in
use.

That would of course require that someone writes the code to bring this
feature!

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html