libcurl - HTTP - The logic for reusing connections and the reasons for closing connections

Hello,

I'm looking for help understanding the way libcurl reuses connections and the reasons for it closing connections which should be reusable for sending HTTP requests
(HTTP 1.1 protocol).
I'm using libcurl in a program for sending many HTTP requests in parallel. I use libcurl with libev, loosely based on the example in https://curl.haxx.se/libcurl/c/evhiperfifo.html.
I use libcurl version 7.65.3.
I use libcurl to send HTTP requests at a constant rate for some minutes (between 8 - 30 minutes). The destination is always an explicit IP address (no DNS resolution needed).
My problem is that during my test runs, libcurl closes connections, and later opens new connections to the same destinations.

I set option CURLMOPT_MAXCONNECTS with a very large number, and so connections don't close because of a limitation on the max size of the connection's cache.
In a setup with Windows OS, I send ~60 HTTP requests per second, all to the same constant IP address. Even though the load stays constant for minutes, libcurl occasionally
closes a connection because it hasn't been used for too long (in function "conn_maxage"), and shortly afterwards it opens another connection to the same destination.
I know libcurl has multiple connections open towards the same destination due to the load. How come it doesn't use one of them for almost 2 minutes, leading
to its closure? libcurl did send many more requests during the 2 minutes than the number of open connections. I assumed that libcurl chooses which connection to reuse
in a round-robin manner. Is it not so? How libcurl chooses which of the open connections to use when it sends a request? Is this behavior influenced by some CURL option? Can someone please direct me to the code where libcurl chooses a free connection from the cache for sending a request?

In another setup with Linux OS, I need to send hundreds (e.g. 600) of HTTP requests per second. There are 10 destination IP addresses. I send requests to them
in a roughly balanced manner. libcurl also closes connections in this case; this time, it does it both from function "conn_maxage", and from function "multi_done"
(("conn->bits.close" is true, although I didn't use an option to prevent libcurl from reusing connections, and
"(premature && !(conn->handler->flags & PROTOPT_STREAM))" is true). I read the comments describing these conditions, but I don't really understand...
Can someone please explain why libcurl closes the connection in this case?

Thanks,
Dorit

This email and any files transmitted with it are confidential material. They are intended solely for the use of the designated individual or entity to whom they are addressed. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, use, distribution or copying of this communication is strictly prohibited and may be unlawful.

If you have received this email in error please immediately notify the sender and delete or destroy any copy of this message

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2020-06-07