Re: Easy Reuse across Multiple Multi Handles

From: Damon Earp via curl-library <curl-library_at_cool.haxx.se> Date: Wed, 29 Jul 2020 18:08:19 -0500

---
libcurl v7.67.0, using a single CURLM and ~400 HTTP/1.1 GET requests:
Setting CURLMOPT_MAX_HOST_CONNECTIONS to 16 lead to > 30 Connections being
used to complete the task. And most of them lived for less than a tenth of
a second.
Setting CURLMOPT_MAX_HOST_CONNECTIONS to 16 and CURLMOPT_MAXCONNECTS to
4096 lead to only 13 connections being used to complete the task. And they
were ready to be used again, until the server timed them out for being idle
after 30 seconds.
---
After I sent that email I upped the parallelism to ~100 requests at a time
and the total number of requests to ~4k. With  CURLMOPT_MAXCONNECTS set the
program only used 16 connections to make all 4k requests. This is what I
wanted, the CURLM to have upto 16 connections established and constantly
being reused. When I don't set maxconnects > 16 connections will be used,
potentially two times more (more testing is needed to determine if that is
true).
In the past few months we moved from v7.61.1 to v7.67.0, which is why I
looked at that briefly. Anecdotally it looks like the newer version of
libcurl is causing more single use connections than the older version. But
as I said, I haven't done a full analysis to determine if that's true.
I did use CURLOPT_VERBOSE initially when trying to figure out if it was
libcurl or my server that was causing the connections to be closed. And
when I saw lines saying "closing as the cache is full" (I don't recall the
exact wording) I started playing with the CURLMOPT_* options.
I also read the man page for CURLMOPT_MAXCONNECTIONS, several times. But
given the behavior I was seeing vs what that was saying made me wonder if
something was amiss. It sounds like the default behavior should be
fine, but my servers are throwing away TCP connections left and right, and
setting it to a large value eliminates the thrashing.
Does that help clear things up?
Damon
On Tue, Jul 28, 2020 at 5:29 PM Daniel Stenberg <daniel_at_haxx.se> wrote:
> On Tue, 28 Jul 2020, Damon Earp wrote:
>
> > I finally got time to sit down and try to figure out what is happening.
> The
> > issue goes away when I set CURLMOPT_MAXCONNECTS. Before I was just
> setting
> > CURLMOPT_MAX_HOST_CONNECTIONS and most of my connections would live for
> a
> > few milliseconds, once MAXCONNECTS is set my reuse and connection
> duration
> > increases.
>
> I read this several times but I didn't understand the conclusion nor the
> metrics shown.
>
> The default size of the connection pool when using the multi interface is
> described in the CURLMOPT_MAXCONNECTS man page (4 times the number of
> added
> easy handles basically). If you want a larger, you of course need to set
> that
> option to something higher.
>
> If you enable verbose mode, libcurl will tell when it re-uses connections
> for
> transfers.
>
> --
>
>   / daniel.haxx.se | Commercial curl support up to 24x7 is available!
>                    | Private help, bug fixes, support, ports, new features
>                    | https://www.wolfssl.com/contact/
>