curl / Mailing Lists / curl-library / Single Mail


Re: Multiple DNS resolution threads are spawned for same host/port for a multi-handle

From: Daniel Stenberg <>
Date: Wed, 22 Mar 2017 23:14:45 +0100 (CET)

On Wed, 22 Mar 2017, mandar choure wrote:

> - 21 Easy handles added to the multi-handle
> We are observing CPU utilization to 100% under load testing when libcurl
> started to spawn multiple threads to resolve DNS. In our case we are using
> same HTTP url and port to post the curl request.

I don't even think starting 21 new threads should use 100% CPU for very long.
Name resolving should be mostly waiting for the first response and the
subsequent calls should be awfully quick - when the OS resolver has the
entries cached for the same names.

Are you absolutely sure it is related to name resolving?

> So it should not spawn multiple threads for DNS resolution for same url. In
> our case we are observing the DNS resolution is taking some time and during
> that interval if another requests comes in it creates another threads and
> thread count increases by 30 under load.

Yeah, this is a known area suitable for improvement as mentioned in the TODO
file here:

> Question is, what is the best way to handle this problem.

Pretty much as the TODO item above says:

  If a name resolve has been initiated for name NN and a second easy handle
  wants to resolve that name as well, make it wait for the first resolve to end
  up in the cache instead of doing a second separate resolve.

... but since it doesn't work like that now, we need to write code that makes
libcurl act like this!

Received on 2017-03-22