On Wed, 22 Mar 2017, mandar choure wrote:

> - 21 Easy handles added to the multi-handle
>
> We are observing CPU utilization to 100% under load testing when libcurl
> started to spawn multiple threads to resolve DNS. In our case we are using
> same HTTP url and port to post the curl request.

I don't even think starting 21 new threads should use 100% CPU for very long.
Name resolving should be mostly waiting for the first response and the
subsequent calls should be awfully quick - when the OS resolver has the
entries cached for the same names.

Are you absolutely sure it is related to name resolving?

> So it should not spawn multiple threads for DNS resolution for same url. In
> our case we are observing the DNS resolution is taking some time and during
> that interval if another requests comes in it creates another threads and
> thread count increases by 30 under load.

Yeah, this is a known area suitable for improvement as mentioned in the TODO
file here:

   https://curl.haxx.se/docs/todo.html#Better_support_for_same_name_res

> Question is, what is the best way to handle this problem.

Pretty much as the TODO item above says:

  If a name resolve has been initiated for name NN and a second easy handle
  wants to resolve that name as well, make it wait for the first resolve to end
  up in the cache instead of doing a second separate resolve.

... but since it doesn't work like that now, we need to write code that makes
libcurl act like this!

-- 
  / daniel.haxx.se
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html