On Thu, Nov 27, 2014 at 12:01:24PM +0000, Josh Russell wrote:
> Hi All,
>
> I'm working on an application which uses a combination of libcurl 7.36.0 and
> c-ares 1.10.1 (c-ares for async DSN lookup). The application is cross compiled
> for various platforms but is closely POSIX like.
>
> I'm using curl_easy_init & curl_easy_setopt to set up my curl handles. Those
> handles are then processed using curl_multi but I'm also using curl_share
> with CURL_LOCK_DATA_COOKIE and CURL_LOCK_DATA_SSL_SESSION.
>
> What I'm finding is that when I make a few requests to various URLs on the same
> domain multiple DNS requests are made for the same domain. Once the first DNS
> response returned its cached I don't see any more requests for that domain - as
> expected.
>
> The problem is, if I make enough requests to the same domain before the first
> DNS response returns, I can my client banned by the DNS server if I make too
> many requests.
>
> Is there any way to de-duplicate concurrent requests for the same DNS address?

I'd be interested in a fix for this as well -- it seems that the DNS
cache has no protection against stampedes when using asynchronous
resolution. This behavior occurs for the threaded resolver, not just
c-ares. Here's a minimal example which can be used to show the
difference:

https://gist.github.com/falconindy/9e5c5de3f0a4e9a871a3

When run with the threaded resolver, strace shows a DNS request for each
handle, i.e. (output trimmed):

$ strace -fe sendmmsg ./resolv
[pid 3536] sendmmsg(5, {{{msg_name(0)=NULL, msg_iov(1)=[{"w}\1\0\0\1\0\0\0\0\0\0\3www\6google\3com\0\0\1\0\1", ...
[pid 3535] sendmmsg(3, {{{msg_name(0)=NULL, msg_iov(1)=[{"\303\207\1\0\0\1\0\0\0\0\0\0\3www\6google\3com\0\0\1\0\1", ...
[pid 3534] sendmmsg(4, {{{msg_name(0)=NULL, msg_iov(1)=[{"\367+\1\0\0\1\0\0\0\0\0\0\3www\6google\3com\0\0\1\0\1", ...
[pid 3533] sendmmsg(6, {{{msg_name(0)=NULL, msg_iov(1)=[{"\313\4\1\0\0\1\0\0\0\0\0\0\3www\6google\3com\0\0\1\0\1", ...

And when preloading a libcurl.so which is built with the vanilla system
resolver, it's just a single request:

$ LD_PRELOAD=~/code/curl/lib/.libs/libcurl.so strace -e sendmmsg ./resolv
sendmmsg(3, {{{msg_name(0)=NULL, msg_iov(1)=[{"\21\37\1\0\0\1\0\0\0\0\0\0\3www\6google\3com\0\0\1\0\1", ...

I'm currently working on porting an application using pthreads and
individual easy handles to using the multi interface and was surprised
to see this behavior. It actually becomes favorable to use synchronous
resolution when many handles are added at once and all connect to the
same domain.

cheers,
dR
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2014-11-27