On Fri, 4 Dec 2015, David Galeano wrote:

> Is there a way to limit the number of new connections opened per second? I
> know I can limit the maximum number of total connections, and the number of
> connections that stay opened on the connection cache, but I have not found
> an easy way to control how many new connections are opened per second.

Right. I don't think anyone has had that problem, or at least hasn't been able
to work-around it with one of the existing options or by just lowering the
pace the application adds new easy handles.

> The reason I am asking this question is because I have problems with some
> proxy servers timing out if I open too many connections too quickly. I can
> open lots of connections slowly, and once open they handle lots of requests
> in parallel without problems, but the proxies seem to collapse when I open
> too many connections in a sort period of time.

I guess they could also collapse as a sort of flood protection or otherwise
get labeled as malicious if the rate goes above some certain threshold from
the same host.

> My program has to run for long periods of time and the load is
> unpredictable, it can be low for a while and suddenly spike with a big
> number of request to perform. So far I have not managed to accurately
> predict how many connections libcurl will open under the hood. I am using
> libcurl's multi interface and I can limit how many handles I add per second,
> but whether those will use existing connections or open new ones is hard to
> predict and to limit.

As we already have an internal queuing system to handle for example
CURLMOPT_MAX_HOST_CONNECTIONS and CURLMOPT_MAX_TOTAL_CONNECTIONS, I don't
think it would be overly complicated to add another option like
CURLMOPT_MAX_CONNECT_RATE or something to perhaps set the maximum number of
connections per second. Would it be enough with just such a knob for "max
rate" or would we (you) need it to be more fancy?

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html