Re: libcurl - HTTP - The logic for reusing connections and the reasons for closing connections

On Sun, 7 Jun 2020, Dorit Mari via curl-library wrote:

> I know libcurl has multiple connections open towards the same destination
> due to the load. How come it doesn't use one of them for almost 2 minutes,
> leading to its closure?

Well either its a bug in curl, or it was sufficient to use the other
connections.

> I assumed that libcurl chooses which connection to reuse in a round-robin
> manner. Is it not so?

No it doesn't. Connections added to the cache will remain in the same order
for as long as they're in there and the cache is searched through in the same
order the next time when it wants to use a connection again.

> How libcurl chooses which of the open connections to use when it sends a
> request?

It basically figures out what connections that could be used, then iterates
over the candidates until one is deemed fine to use.

> Is this behavior influenced by some CURL option?

No. Unless you count CURLOPT_MAXAGE_CONN, which is the number of seconds a
connection is allowed to be idle and still remain in the connection cache (118
by default).

> Can someone please direct me to the code where libcurl chooses a free
> connection from the cache for sending a request?

lib/url.c:ConnectionExists() - but buckle up before you dive in, it's 400
lines with decades of legacy and not one of the functions in libcurl I'm most
proud of...

> In another setup with Linux OS, I need to send hundreds (e.g. 600) of HTTP
> requests per second. There are 10 destination IP addresses. I send requests
> to them in a roughly balanced manner. libcurl also closes connections in
> this case; this time, it does it both from function "conn_maxage", and from
> function "multi_done" (("conn->bits.close" is true, although I didn't use an
> option to prevent libcurl from reusing connections, and "(premature &&
> !(conn->handler->flags & PROTOPT_STREAM))" is true). I read the comments
> describing these conditions, but I don't really understand... Can someone
> please explain why libcurl closes the connection in this case?

Without more details it is impossible to know. Perhaps because the server said
Connection: close ? Perhaps something else. libcurl makes an effort to never
close connections it doesn't need to.

> This email and any files transmitted with it are confidential material.

Consider the confidential material to have leaked.

-- 
  / daniel.haxx.se | Commercial curl support up to 24x7 is available!
                   | Private help, bug fixes, support, ports, new features
                   | https://www.wolfssl.com/contact/
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html