Re: Easy Reuse across Multiple Multi Handles

Sorry, I'm not claiming there's a bug. I was expecting that by limiting the
max connections to a host those few connections would be reused heavily.
For example 100 requests all going to 1 host with a MAX_HOST_CONNECTIONS of
10 would be 10 connections with an average of 10 requests each. My
expectation was wrong, instead I saw a lot of single use connections, each
TCP and TLS setup and teardown to do 1/100th of the above work not 1/10. By
disabling the MAXCONNECTS default algorithm, I get the behavior I was
looking for.

I was trying to not waste your time with the nitty gritty, but as you are
asking, I guess I left too much context out. I am setting an upper bound in
the multi handle to limit the number of concurrent requests between the
servers, so they cannot swarm each other. Furthermore these computers are
constantly talking with one another and the tcp/tls connect time is
measurable. After adding the CURLMOPT_MAXCONNECTS=N we reduced the number
of TIME_WAIT sockets on our nodes by a factor of 10.

Thanks for the great tool and taking the time to respond to these emails.

On Thu, Jul 30, 2020 at 5:11 AM Daniel Stenberg <daniel_at_haxx.se> wrote:

> On Wed, 29 Jul 2020, Damon Earp via curl-library wrote:
>
> > In the past few months we moved from v7.61.1 to v7.67.0, which is why I
> > looked at that briefly. Anecdotally it looks like the newer version of
> > libcurl is causing more single use connections than the older version.
> But
> > as I said, I haven't done a full analysis to determine if that's true.
>
> Things that will impact the behavior that you didn't mention (or perhaps I
> missed it):
>
> Parallelism - if you add 100 easy handles to a multi handle, they will be
> done
> mostly in parallel and thus they can't reuse connections.
>
> CURLMOPT_MAX_HOST_CONNECTIONS will limit the parallelism and will libcurl
> rather queue transfers internally if you reach the host name limit.
>
> You didn't say how many different host names you're using nor how many
> transfers in parallel you're doing - but since you refer to the connection
> cache being full, then I think that you simply run into connection pool
> size
> issues perhaps because you use a large number of host names.
>
> If you think this is due to some actual problem or bug in libcurl, then I
> would need more specific instructions on how to reproduce.
>
> --
>
> / daniel.haxx.se | Commercial curl support up to 24x7 is available!
> | Private help, bug fixes, support, ports, new features
> | https://www.wolfssl.com/contact/
>

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2020-07-31