cURL / Mailing Lists / curl-library / Single Mail

curl-library

RE: Many CLOSE_WAIT when handling lots of URLs

From: Daniel Stenberg <daniel_at_haxx.se>
Date: Wed, 12 Feb 2014 10:10:59 +0100 (CET)

On Wed, 12 Feb 2014, Shao, Shuchao wrote:

>> That's plain incorrect. Read the code again.

> Read the latest 7.34.0 again, the function ConnectionDone() use
> multi->maxconnects to decide if the connection cache is full.

Right. So the value _is_ used somewhere!

> But this value is 0 by default, so there is no limit for the cache
> connections number by default.

Default 0 would be a bug! The documentation says:

   "Default is 10, and libcurl will enlarge the size for each added easy handle
   to make it fit 4 times the number of added easy handles. "

So there's definately a fix needed here. The documentation could be improved
too as it seems stupid to say that the default is 10 but as soon as you add
one handle, it'll be 4...

> In fact, if I use curl_multi_setopt() set the value CURLMOPT_MAXCONNECTS as
> a non-zero value, like as 10, my program will be much better, the CLOSE_WAIT
> number will accumulate more slow. However, it is still accumulating
> CLOSE_WAIT. If I print the values of multi->maxconnects and
> state.conn_cache->num_connections in the function ConnectionDone(), the
> value of state.conn_cache->num_connections keep growing even if it has been
> greater than multi->maxconnects (the growing speed is slower than when
> setting multi->maxconnects as default value 0). So the CURLMOPT_MAXCONNECTS
> option does not take effect.

But how many handles have you added then? The maxconnects setting does not
override the number of handles you add.

Can I also remind you that you still haven't provided any good example for us
to repeat this problem so we keep having this discussion where you state vague
things, I see the opposite in the code and we continue without progress. It is
a highly unproductive way of fixing a problem.

Can we instead change this to me asking you for a patch against git HEAD that
makes sure maxconnects works as documented? It probably won't address all the
CLOSE_WAIT connections you see but it should definately limit them and once
that works we can continue to look and see if there are additional problems
that may explain any further strange behaviors.

> BTW, in commit d021f2e8a00, there is no this decision on maxconnects in
> ConnectionDone(), so I said CURLMOPT_MAXCONNECTS is not used in libcurl.

And again, I don't particularly care about how things worked back then. I
live and work in the present time and I'm interested in how the current code
works and what we should do in the current code to improve things.

You're free to do whatever you want with the old code base but don't expect me
to go there with you.

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Received on 2014-02-12