Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
RE: Question about DNS timeout in libCurl
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>
Date: Mon, 13 Dec 2021 20:56:45 +0000
> Because we prefer ares_getaddrinfo() now and then c-ares gets both A and AAAA addresses by itself.
Somehow, I didn't find ares_getaddrinfo() in libcurl code, so you probably meant ares_gethostbyname() used by Curl_resolver_getaddrinfo() in lib\asyn-ares.c, right?
> If we believe it would cause issues when lowered in c-ares, why will it not cause issues if lowered in libcurl? To me, it seems like a value that should be safe to lower in both or in neither...
Yes, I would also rather lower the timeout in both c-ares and libcurl, but I guess the current c-ares 5s timeout comes from the default DNS timeout value from resolv.conf and Linux.
The https://man7.org/linux/man-pages/man5/resolver.5.html says that it is 5s, and because c-ares ignores the "timeout" option in the resolv.conf,
it essentially always uses the Linux default value if DNS timeout is not specified explicitly.
So, my concern was that if we lowered it in c-ares, then it would not be aligned with the default 5s value expected from the resolv.conf on Linux systems anymore, which might be not a default behavior expected from a general-purpose resolver library.
But on the other hand, libcurl doesn't have to follow exactly the resolver library constraints (if it is agreed that having 5s default in c-ares to match the default value from resolv.conf is a must ), and it can use its own default DNS resolver value, which may be lower than the c-ares default value.
> The order and preference I would like to see this handled:
>
> 1. Make c-ares use a better default timeout or method to avoid long waits,
> possibly by doing more in parallel
> 2. Find and use a better default timeout for c-ares
> 3. Find and use a better default c-ares timeout set by libcurl
> 4. Only if the above solutions cannot be made to work, I think we should
> consider allowing this to be set by an application using libcurl.
If it is agreed that changing the c-ares default timeout from 5s to some lower value and thus breaking its connection with the default value implied by resolv.conf is not a problem, then it is the best plan.
Especially, if things can be done in parallel, which is very important for dual-stack and Happy Eyeballs.
But with parallel approach, there is also some problem, because resolv.conf also expects the resolver to process name servers sequentially, so there probably should be some additional option in c-ares allowing to break this assumption and perform parallel queries in parallel (or by default only run parallel queries for IPv/IPv6 queues ).
And implementing this approach in c-ares is obviously not a trivial thing and not for the nearest future.
So, if the default c-ares 5s timeout must stay (based on the considerations above or by some other reasons), then we probably have no other choice besides setting the DNS timeout for c-ares in libcurl explicitly.
And maybe the simplest short-term solution is to have a preprocessor constant specifying curl default DNS timeout and use it in Curl_resolver_init() to set c-ares timeout (and make HAPPY_EYEBALLS_DNS_TIMEOUT equal to it as well).
We can set it to either 5s to match the current c-ares default timeout to avoid any surprises, or to some lower values.
This approach will save the trouble of creating and exposing a new handle option, but it will allow to have different default DNS timeouts in some specific libcurl builds if some applications urgently need them.
Thanks,
Dmitry Karpov
-----Original Message-----
From: Daniel Stenberg <daniel_at_haxx.se>
Sent: Saturday, December 11, 2021 3:34 AM
To: Dmitry Karpov <dkarpov_at_roku.com>
Cc: Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>
Subject: RE: Question about DNS timeout in libCurl
On Fri, 10 Dec 2021, Dmitry Karpov wrote:
>> that timeout logic in curl is also only used when an old c-ares
>> version is used
>
> Could you please clarify the last point about the old c-ares version?
> I stepped on this issue using c-ares 1.17.1, which I think it is not that old.
Because we prefer ares_getaddrinfo() now and then c-ares gets both A and AAAA addresses by itself.
> I agree with you that it is mostly a c-ares problem but lowering the
> default timeout in c-ares itself may create sudden issues in the apps
> that rely on this value.
If we believe it would cause issues when lowered in c-ares, why will it not cause issues if lowered in libcurl? To me, it seems like a value that should be safe to lower in both or in neither...
> The current libcurl version hard codes this value and relies on this
> as a c-ares documented feature, so I doubt that the default c-ares
> timeout will change in the future to avoid issues with existing apps.
The timeout that isn't used by current libcurl with current c-ares you mean.
But libcurl can configure c-ares to use a specific timeout that's right.
> So, the most logical way to decrease the default DNS timeout used in
> libcurl seems to implement some logic controlling timing of DNS
> queries. In case of c-ares used as a DNS resolver, such logic may use
> c-ares API for setting the timeout, for other resolvers it may use
> something else or just rely on the defaults.
The order and preference I would like to see this handled:
1. Make c-ares use a better default timeout or method to avoid long waits,
possibly by doing more in parallel
2. Find and use a better default timeout for c-ares
3. Find and use a better default c-ares timeout set by libcurl
4. Only if the above solutions cannot be made to work, I think we should
consider allowing this to be set by an application using libcurl.
Date: Mon, 13 Dec 2021 20:56:45 +0000
> Because we prefer ares_getaddrinfo() now and then c-ares gets both A and AAAA addresses by itself.
Somehow, I didn't find ares_getaddrinfo() in libcurl code, so you probably meant ares_gethostbyname() used by Curl_resolver_getaddrinfo() in lib\asyn-ares.c, right?
> If we believe it would cause issues when lowered in c-ares, why will it not cause issues if lowered in libcurl? To me, it seems like a value that should be safe to lower in both or in neither...
Yes, I would also rather lower the timeout in both c-ares and libcurl, but I guess the current c-ares 5s timeout comes from the default DNS timeout value from resolv.conf and Linux.
The https://man7.org/linux/man-pages/man5/resolver.5.html says that it is 5s, and because c-ares ignores the "timeout" option in the resolv.conf,
it essentially always uses the Linux default value if DNS timeout is not specified explicitly.
So, my concern was that if we lowered it in c-ares, then it would not be aligned with the default 5s value expected from the resolv.conf on Linux systems anymore, which might be not a default behavior expected from a general-purpose resolver library.
But on the other hand, libcurl doesn't have to follow exactly the resolver library constraints (if it is agreed that having 5s default in c-ares to match the default value from resolv.conf is a must ), and it can use its own default DNS resolver value, which may be lower than the c-ares default value.
> The order and preference I would like to see this handled:
>
> 1. Make c-ares use a better default timeout or method to avoid long waits,
> possibly by doing more in parallel
> 2. Find and use a better default timeout for c-ares
> 3. Find and use a better default c-ares timeout set by libcurl
> 4. Only if the above solutions cannot be made to work, I think we should
> consider allowing this to be set by an application using libcurl.
If it is agreed that changing the c-ares default timeout from 5s to some lower value and thus breaking its connection with the default value implied by resolv.conf is not a problem, then it is the best plan.
Especially, if things can be done in parallel, which is very important for dual-stack and Happy Eyeballs.
But with parallel approach, there is also some problem, because resolv.conf also expects the resolver to process name servers sequentially, so there probably should be some additional option in c-ares allowing to break this assumption and perform parallel queries in parallel (or by default only run parallel queries for IPv/IPv6 queues ).
And implementing this approach in c-ares is obviously not a trivial thing and not for the nearest future.
So, if the default c-ares 5s timeout must stay (based on the considerations above or by some other reasons), then we probably have no other choice besides setting the DNS timeout for c-ares in libcurl explicitly.
And maybe the simplest short-term solution is to have a preprocessor constant specifying curl default DNS timeout and use it in Curl_resolver_init() to set c-ares timeout (and make HAPPY_EYEBALLS_DNS_TIMEOUT equal to it as well).
We can set it to either 5s to match the current c-ares default timeout to avoid any surprises, or to some lower values.
This approach will save the trouble of creating and exposing a new handle option, but it will allow to have different default DNS timeouts in some specific libcurl builds if some applications urgently need them.
Thanks,
Dmitry Karpov
-----Original Message-----
From: Daniel Stenberg <daniel_at_haxx.se>
Sent: Saturday, December 11, 2021 3:34 AM
To: Dmitry Karpov <dkarpov_at_roku.com>
Cc: Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>
Subject: RE: Question about DNS timeout in libCurl
On Fri, 10 Dec 2021, Dmitry Karpov wrote:
>> that timeout logic in curl is also only used when an old c-ares
>> version is used
>
> Could you please clarify the last point about the old c-ares version?
> I stepped on this issue using c-ares 1.17.1, which I think it is not that old.
Because we prefer ares_getaddrinfo() now and then c-ares gets both A and AAAA addresses by itself.
> I agree with you that it is mostly a c-ares problem but lowering the
> default timeout in c-ares itself may create sudden issues in the apps
> that rely on this value.
If we believe it would cause issues when lowered in c-ares, why will it not cause issues if lowered in libcurl? To me, it seems like a value that should be safe to lower in both or in neither...
> The current libcurl version hard codes this value and relies on this
> as a c-ares documented feature, so I doubt that the default c-ares
> timeout will change in the future to avoid issues with existing apps.
The timeout that isn't used by current libcurl with current c-ares you mean.
But libcurl can configure c-ares to use a specific timeout that's right.
> So, the most logical way to decrease the default DNS timeout used in
> libcurl seems to implement some logic controlling timing of DNS
> queries. In case of c-ares used as a DNS resolver, such logic may use
> c-ares API for setting the timeout, for other resolvers it may use
> something else or just rely on the defaults.
The order and preference I would like to see this handled:
1. Make c-ares use a better default timeout or method to avoid long waits,
possibly by doing more in parallel
2. Find and use a better default timeout for c-ares
3. Find and use a better default c-ares timeout set by libcurl
4. Only if the above solutions cannot be made to work, I think we should
consider allowing this to be set by an application using libcurl.
-- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.htmlReceived on 2021-12-13