cURL / Mailing Lists / curl-library / Single Mail

curl-library

RE: Many CLOSE_WAIT when handling lots of URLs

From: Shao, Shuchao <sshao_at_websense.com>
Date: Thu, 20 Feb 2014 02:48:39 +0000

> You make it sound as if our job is primarily to limit the number of
> CLOSE_WAIT connections. It really isn't.
>
> The CLOSE_WAIT connections are mostly (AFAIK) unfortunate side-effects of
> us keeping connections alive after we're done using them, and while kept
> like that they get closed from the other end.
 
Many CLOSE_WAIT connections will impact the libcurl performance significantly. Every system has a limit for maximum number of open socket, the more CLOSE_WAIT connection will result in less ESTABLISHED connections.

> I'm thinking we can change the if() in ConnectionDone() to a while() instead
> and kill off connections until below the threshold. We should also add a test
> case for that.

Just changing if() to while() seems does not have a lot help.
In my setup,
Using libcurl-7.24.0, I can have about 20K ESTABLISHED connections, no CLOSE_WAIT connections. Using 7.35.0 (after changing if() to while()), the number will be 9K for ESTABLISHED and 11K for CLOSE_WAIT.

I am wondering if we need to close the connections for a easy_handle when we removing it from a multi_handle?

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2014-02-20