Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Easy Reuse across Multiple Multi Handles
From: Damon Earp via curl-library <curl-library_at_cool.haxx.se>
Date: Wed, 29 Jul 2020 18:08:19 -0500
Date: Wed, 29 Jul 2020 18:08:19 -0500
I'll try to rephrase:
--- libcurl v7.67.0, using a single CURLM and ~400 HTTP/1.1 GET requests: Setting CURLMOPT_MAX_HOST_CONNECTIONS to 16 lead to > 30 Connections being used to complete the task. And most of them lived for less than a tenth of a second. Setting CURLMOPT_MAX_HOST_CONNECTIONS to 16 and CURLMOPT_MAXCONNECTS to 4096 lead to only 13 connections being used to complete the task. And they were ready to be used again, until the server timed them out for being idle after 30 seconds. --- After I sent that email I upped the parallelism to ~100 requests at a time and the total number of requests to ~4k. With CURLMOPT_MAXCONNECTS set the program only used 16 connections to make all 4k requests. This is what I wanted, the CURLM to have upto 16 connections established and constantly being reused. When I don't set maxconnects > 16 connections will be used, potentially two times more (more testing is needed to determine if that is true). In the past few months we moved from v7.61.1 to v7.67.0, which is why I looked at that briefly. Anecdotally it looks like the newer version of libcurl is causing more single use connections than the older version. But as I said, I haven't done a full analysis to determine if that's true. I did use CURLOPT_VERBOSE initially when trying to figure out if it was libcurl or my server that was causing the connections to be closed. And when I saw lines saying "closing as the cache is full" (I don't recall the exact wording) I started playing with the CURLMOPT_* options. I also read the man page for CURLMOPT_MAXCONNECTIONS, several times. But given the behavior I was seeing vs what that was saying made me wonder if something was amiss. It sounds like the default behavior should be fine, but my servers are throwing away TCP connections left and right, and setting it to a large value eliminates the thrashing. Does that help clear things up? Damon On Tue, Jul 28, 2020 at 5:29 PM Daniel Stenberg <daniel_at_haxx.se> wrote: > On Tue, 28 Jul 2020, Damon Earp wrote: > > > I finally got time to sit down and try to figure out what is happening. > The > > issue goes away when I set CURLMOPT_MAXCONNECTS. Before I was just > setting > > CURLMOPT_MAX_HOST_CONNECTIONS and most of my connections would live for > a > > few milliseconds, once MAXCONNECTS is set my reuse and connection > duration > > increases. > > I read this several times but I didn't understand the conclusion nor the > metrics shown. > > The default size of the connection pool when using the multi interface is > described in the CURLMOPT_MAXCONNECTS man page (4 times the number of > added > easy handles basically). If you want a larger, you of course need to set > that > option to something higher. > > If you enable verbose mode, libcurl will tell when it re-uses connections > for > transfers. > > -- > > / daniel.haxx.se | Commercial curl support up to 24x7 is available! > | Private help, bug fixes, support, ports, new features > | https://www.wolfssl.com/contact/ >
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2020-07-30