Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
RE: Question about DNS timeout in libCurl
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>
Date: Thu, 16 Dec 2021 20:07:27 +0000
> How do other common getaddrinfo implementations handle (timeouts for) non-responding name reservers? It seems like behavior we should be able to mimick.
The regular getaddrinfo() on Linux systems follows the resolv.conf spec and uses 5s timeout by default when switching from one name server to the other.
C-ares currently follows the same convention, so both c-ares and getaddrinfo() provide the same expected behavior if name servers in the resolv.conf don't use the 'timeout' option.
The 'timeout' name server option in the resolv.conf is supposed to specify a timeout for some specific name server, but c-ares currently ignores it.
That's probably where c-ares can also improve and start honoring this option in the future releases.
> If we can't improve c-ares to do this better then I think this is a change to consider, yes. I want us to first explore fixing this in the resolver code.
Yes, c-ares definitely can be improved, but I think that it can't probably ignore the resolv.conf spec and change the default 5s timeout imposed by the spec.
Doing so will create a difference between regular getaddrinfo() and c-ares behaviors, be against the expected resolver behavior and can be considered as c-ares regression from the apps that rely on resolver compliance with the resolv.conf spec.
So, c-ares team may have all good reasons to reject the idea of decreasing the default timeout, I am afraid, just for the sake of libcurl, arguing that c-ares does provide API to change the timeout if some app doesn't like its default value, but by default they should follow the resolv.conf expectations.
Let me summarize the c-ares improvements which might help libcurl to better handle resolution issues, so we can compile a list of suggestions/feature requests for the c-ares mailing list.
1. Honor 'timeout' option in the resolv.conf.
- This will make c-ares fully compliant with the resolv.conf spec and allow to specify DNS timeouts for some specific name servers on a system level.
This may be enough to work around some DNS timeout issues if some project has control over resolv.conf
2. Provide c-ares option to run name resolution queries for different name servers in parallel instead of doing it sequentially, which should help to find a good name resolution much faster.
One flavor of this approach, if it can help to simplify things, can be running queues for IPv4 and IPv6 name servers in parallel, while iterating each family sequentially - thus following Happy Eyeballs philosophy.
These changes will keep the default c-ares timeout (as resolv.conf prescribes) but should let libcurl to better handle problems with name servers without doing some explicit timeout manipulation stuff.
Please comment/add more to the suggestions, and I will put them into the c-ares mailing list.
Thanks,
Dmitry Karpov
-----Original Message-----
From: Daniel Stenberg <daniel_at_haxx.se>
Sent: Thursday, December 16, 2021 12:05 AM
To: Dmitry Karpov <dkarpov_at_roku.com>
Cc: Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>
Subject: RE: Question about DNS timeout in libCurl
On Tue, 14 Dec 2021, Dmitry Karpov wrote:
> But the question remains whether libcurl should always rely on a
> default DNS timeout set by c-ares, whatever it is, instead of
> explicitly setting the "curl" one?
We also don't set any timeouts for the stock name resolver functions but rely on their internally used timeouts. I don't think curl has better or more information than the resolver library so I don't think we can make a better decision.
How do other common getaddrinfo implementations handle (timeouts for) non-responding name reservers? It seems like behavior we should be able to mimick.
> The concern is that different c-ares versions may have different
> default timeouts which can create inconsistent behavior
Yes, but that's more of a feature than a bug. When c-ares fixes problems in later versions it is a good thing that curl can work with them. "inconsistent behavior" goes both ways.
> makes tight coupling libcurl with c-ares default timeout.
Yes, but now we're back in "old code" territory and that old way of using c-ares had more problem than just that timer dependency so I suggest you move away from that. The "new code" path sets no timeout at all but relies on c-ares' default.
> options.timeout = DEFAULT_DNS_TIMEOUT;
> This simple change will make libcurl DNS timeout consistent regardless
> if the default timeout changes in c-ares.
If we can't improve c-ares to do this better then I think this is a change to consider, yes. I want us to first explore fixing this in the resolver code.
Date: Thu, 16 Dec 2021 20:07:27 +0000
> How do other common getaddrinfo implementations handle (timeouts for) non-responding name reservers? It seems like behavior we should be able to mimick.
The regular getaddrinfo() on Linux systems follows the resolv.conf spec and uses 5s timeout by default when switching from one name server to the other.
C-ares currently follows the same convention, so both c-ares and getaddrinfo() provide the same expected behavior if name servers in the resolv.conf don't use the 'timeout' option.
The 'timeout' name server option in the resolv.conf is supposed to specify a timeout for some specific name server, but c-ares currently ignores it.
That's probably where c-ares can also improve and start honoring this option in the future releases.
> If we can't improve c-ares to do this better then I think this is a change to consider, yes. I want us to first explore fixing this in the resolver code.
Yes, c-ares definitely can be improved, but I think that it can't probably ignore the resolv.conf spec and change the default 5s timeout imposed by the spec.
Doing so will create a difference between regular getaddrinfo() and c-ares behaviors, be against the expected resolver behavior and can be considered as c-ares regression from the apps that rely on resolver compliance with the resolv.conf spec.
So, c-ares team may have all good reasons to reject the idea of decreasing the default timeout, I am afraid, just for the sake of libcurl, arguing that c-ares does provide API to change the timeout if some app doesn't like its default value, but by default they should follow the resolv.conf expectations.
Let me summarize the c-ares improvements which might help libcurl to better handle resolution issues, so we can compile a list of suggestions/feature requests for the c-ares mailing list.
1. Honor 'timeout' option in the resolv.conf.
- This will make c-ares fully compliant with the resolv.conf spec and allow to specify DNS timeouts for some specific name servers on a system level.
This may be enough to work around some DNS timeout issues if some project has control over resolv.conf
2. Provide c-ares option to run name resolution queries for different name servers in parallel instead of doing it sequentially, which should help to find a good name resolution much faster.
One flavor of this approach, if it can help to simplify things, can be running queues for IPv4 and IPv6 name servers in parallel, while iterating each family sequentially - thus following Happy Eyeballs philosophy.
These changes will keep the default c-ares timeout (as resolv.conf prescribes) but should let libcurl to better handle problems with name servers without doing some explicit timeout manipulation stuff.
Please comment/add more to the suggestions, and I will put them into the c-ares mailing list.
Thanks,
Dmitry Karpov
-----Original Message-----
From: Daniel Stenberg <daniel_at_haxx.se>
Sent: Thursday, December 16, 2021 12:05 AM
To: Dmitry Karpov <dkarpov_at_roku.com>
Cc: Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>
Subject: RE: Question about DNS timeout in libCurl
On Tue, 14 Dec 2021, Dmitry Karpov wrote:
> But the question remains whether libcurl should always rely on a
> default DNS timeout set by c-ares, whatever it is, instead of
> explicitly setting the "curl" one?
We also don't set any timeouts for the stock name resolver functions but rely on their internally used timeouts. I don't think curl has better or more information than the resolver library so I don't think we can make a better decision.
How do other common getaddrinfo implementations handle (timeouts for) non-responding name reservers? It seems like behavior we should be able to mimick.
> The concern is that different c-ares versions may have different
> default timeouts which can create inconsistent behavior
Yes, but that's more of a feature than a bug. When c-ares fixes problems in later versions it is a good thing that curl can work with them. "inconsistent behavior" goes both ways.
> makes tight coupling libcurl with c-ares default timeout.
Yes, but now we're back in "old code" territory and that old way of using c-ares had more problem than just that timer dependency so I suggest you move away from that. The "new code" path sets no timeout at all but relies on c-ares' default.
> options.timeout = DEFAULT_DNS_TIMEOUT;
> This simple change will make libcurl DNS timeout consistent regardless
> if the default timeout changes in c-ares.
If we can't improve c-ares to do this better then I think this is a change to consider, yes. I want us to first explore fixing this in the resolver code.
-- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.htmlReceived on 2021-12-16