Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: IPv6 resolution problems for IPv4 resolve mode
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Timothe Litt <litt_at_acm.org>
Date: Mon, 11 Jul 2022 18:57:54 -0400
The terminology and behaviors here are conflating two very different things:
- What address types are added to the cache
- What addresses are used for a connection.
The name IPRESOLVE_* implies that it controls resolution, but the
description says it controls "what kind of IP addresses to use when
establishing a (possibly cached) connection:.
Complicating this is the DNS cache, which is shared across all curl
handles. Evicting cache entries to control connections is
counter-intuitive and inefficient. Consider the case of one handle
wanting IPv4 for SSH to a host, but IPv6 for http...
The cache should, subject to TTL, hold all resolutions ever performed
for a name - v4, v6, or both. This includes caching the case of a
resolution returning "Answers:0" (no address of that type).
The connection should be made based on the address types permitted for
its handle - if an allowed address type is not in the cache, and has
never previously been queried, then a query should be triggered and the
result cached.
And I'd deprecate the CURLOPT_IPRESOLVE name, aliasing it to
CURLOPT_CONNECTWITH => CURL_USE_IPV4, CURL_USE_IPV6 (and alias
CURL_IPRESOLVE_WHATEVER to CURL_USE_IPV4 | CURL_USE_IPV6 - which the
current "0" can default to for backwards compatibility).
With respect to a default: it should be both. The fact that some issues
still occur with IPv6 in some environments just means that those
environments are broken. It's fine to have a work-around for them. But
defaulting to the work-around only discourages these from being fixed.
Like it or not, IPv6 is the going-forward plan for the network and
adoption is happening, however slowly.
Again: the handle should specify how to connect. The cache should hold
the result of lookups, which are triggered when needed. Don't conflate
the two.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
On 11-Jul-22 17:16, Dmitry Karpov via curl-library wrote:
> I like the idea of adding additional bit to CURLOPT_IPRESOLVE, but actually I think about the "Auto" or "Dual" mode, which should probably provide the current "both" IPv4 and IPv6 instead of the "Also" mode enforcing certain name resolution (i.e. IPv4),.
>
> And probably the "auto" mode should not be used by default, because when I enabled IPv6 in libcurl, I stepped on name resolution regressions and even timeouts, which didn't occur when IPv6 wasn't used.
> I used explicit CURL_IPRESOLVE_V4 settings to avoid any dual-stack issues, but still stepped on them in a quite unexpected way.
>
> So, I think that results delivered by explicit CURL_IPRESOLVE_V4 should be agnostic by default to whether IPv6 is enabled in libcurl or not.
> It is just doesn't seem natural to specify CURL_IPRESOLVE_V4 to indicate that only IPv4 addresses should be used and step on problems with not relevant IPv6 queries.
>
> That's why I think that by default the behavior should be probably like what my patch provides:
>
> CURL_IPRESOLVE_V4 - only IPv4 DNS resolution is used
> CURL_IPRESOLVE_WHATEVER - both IPv4 and IPv6 DNS resolutions
> CURL_IPRESOLVE_V6 - both IPv4 and IPv6 DNS resolutions
>
> The reason why I think that CURL_IPRESOLVE_V6 should provide both IPv4 and IPv6 name resolutions is because IPv4 is still much better supported than IPv6, and it is more likely to have DNS problems with IPv6 than with IPv4 (at least that's my experience with a quite large set of ISPs and routers).
> This means that if some client wants manually (not via the CURL_IPRESOLVE_WHATEVER dual-stack option) to try IPv6 first and switch to IPv4 in case of failure, then successful IPv4 name resolution will be very handy.
>
>> Your patch looks like it might work. I'm still mostly concerned that this takes us back in behavior a little bit and that it will make someone else not like that and come here in a few months with an issue about it.
> Sure, I understand the concern. But with the IPv6 DNS issues that I observed (slow and missing AAAA responses) the use case described in 84d2839740ca7804 may result into timeouts or slow name resolution regressions.
> With my patch (or some other similar solution), it will be able to avoid such regressions with a price of additional AAAA query when the client decides to switch to IPv6 resolve mode after IPv4.
> This switch to IPv6 will fail under such circumstances, but because IPv4 host name resolution is already in the cache, the client can quickly switch back to IPv4 and proceed the transfer.
>
> And with the "Auto" mode bit turned on, some special clients can enforce "dual" queries even for IPv4 resolve mode, but this seems like a very special use case to me to be enabled by default.
>
> Thanks,
> Dmitry Karpov
>
>
> -----Original Message-----
> From: Daniel Stenberg<daniel_at_haxx.se>
> Sent: Saturday, July 9, 2022 3:14 PM
> To: Dmitry Karpov via curl-library<curl-library_at_lists.haxx.se>
> Cc: Dmitry Karpov<dkarpov_at_roku.com>
> Subject: RE: IPv6 resolution problems for IPv4 resolve mode
>
> On Thu, 7 Jul 2022, Dmitry Karpov via curl-library wrote:
>
>> So, the intent described in the statement "It is important that *all*
>> addresses versions are resolved, even if not used..." from
>> 84d2839740ca7804 will cause transfer timeouts for IPv4-only resolution
>> mode in case of bad/missing AAAA responses.
> I would rather say that it *may* cause timouts, as it certainly is not how things *should* work.
>
> Your patch looks like it might work. I'm still mostly concerned that this takes us back in behavior a little bit and that it will make someone else not like that and come here in a few montths with an issue about it.
>
> Wouldn't it be better if we add a bit to the CURLOPT_IPRESOLVE option that says something like "enforce this in name resolve too" ? Like:
>
> #define CURL_IPRESOLVE_ALSO_RESOLVE 0x10 /* name to be debated */
>
> to be used like this:
>
> curl_easy_setopt(c, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 |
> CURL_IPRESOLVE_ALSO_RESOLVE );
>
> ... which then makes the name resolve for this transfer also just ask for IPv4 and the connect only be done using IPv4?
>
> Maybe even make the DNS cache use family preference as a key for lookups so that we have separate IPv6-only and IPv4-only results in memory for the same host?
>
Received on 2022-07-12
Date: Mon, 11 Jul 2022 18:57:54 -0400
The terminology and behaviors here are conflating two very different things:
- What address types are added to the cache
- What addresses are used for a connection.
The name IPRESOLVE_* implies that it controls resolution, but the
description says it controls "what kind of IP addresses to use when
establishing a (possibly cached) connection:.
Complicating this is the DNS cache, which is shared across all curl
handles. Evicting cache entries to control connections is
counter-intuitive and inefficient. Consider the case of one handle
wanting IPv4 for SSH to a host, but IPv6 for http...
The cache should, subject to TTL, hold all resolutions ever performed
for a name - v4, v6, or both. This includes caching the case of a
resolution returning "Answers:0" (no address of that type).
The connection should be made based on the address types permitted for
its handle - if an allowed address type is not in the cache, and has
never previously been queried, then a query should be triggered and the
result cached.
And I'd deprecate the CURLOPT_IPRESOLVE name, aliasing it to
CURLOPT_CONNECTWITH => CURL_USE_IPV4, CURL_USE_IPV6 (and alias
CURL_IPRESOLVE_WHATEVER to CURL_USE_IPV4 | CURL_USE_IPV6 - which the
current "0" can default to for backwards compatibility).
With respect to a default: it should be both. The fact that some issues
still occur with IPv6 in some environments just means that those
environments are broken. It's fine to have a work-around for them. But
defaulting to the work-around only discourages these from being fixed.
Like it or not, IPv6 is the going-forward plan for the network and
adoption is happening, however slowly.
Again: the handle should specify how to connect. The cache should hold
the result of lookups, which are triggered when needed. Don't conflate
the two.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
On 11-Jul-22 17:16, Dmitry Karpov via curl-library wrote:
> I like the idea of adding additional bit to CURLOPT_IPRESOLVE, but actually I think about the "Auto" or "Dual" mode, which should probably provide the current "both" IPv4 and IPv6 instead of the "Also" mode enforcing certain name resolution (i.e. IPv4),.
>
> And probably the "auto" mode should not be used by default, because when I enabled IPv6 in libcurl, I stepped on name resolution regressions and even timeouts, which didn't occur when IPv6 wasn't used.
> I used explicit CURL_IPRESOLVE_V4 settings to avoid any dual-stack issues, but still stepped on them in a quite unexpected way.
>
> So, I think that results delivered by explicit CURL_IPRESOLVE_V4 should be agnostic by default to whether IPv6 is enabled in libcurl or not.
> It is just doesn't seem natural to specify CURL_IPRESOLVE_V4 to indicate that only IPv4 addresses should be used and step on problems with not relevant IPv6 queries.
>
> That's why I think that by default the behavior should be probably like what my patch provides:
>
> CURL_IPRESOLVE_V4 - only IPv4 DNS resolution is used
> CURL_IPRESOLVE_WHATEVER - both IPv4 and IPv6 DNS resolutions
> CURL_IPRESOLVE_V6 - both IPv4 and IPv6 DNS resolutions
>
> The reason why I think that CURL_IPRESOLVE_V6 should provide both IPv4 and IPv6 name resolutions is because IPv4 is still much better supported than IPv6, and it is more likely to have DNS problems with IPv6 than with IPv4 (at least that's my experience with a quite large set of ISPs and routers).
> This means that if some client wants manually (not via the CURL_IPRESOLVE_WHATEVER dual-stack option) to try IPv6 first and switch to IPv4 in case of failure, then successful IPv4 name resolution will be very handy.
>
>> Your patch looks like it might work. I'm still mostly concerned that this takes us back in behavior a little bit and that it will make someone else not like that and come here in a few months with an issue about it.
> Sure, I understand the concern. But with the IPv6 DNS issues that I observed (slow and missing AAAA responses) the use case described in 84d2839740ca7804 may result into timeouts or slow name resolution regressions.
> With my patch (or some other similar solution), it will be able to avoid such regressions with a price of additional AAAA query when the client decides to switch to IPv6 resolve mode after IPv4.
> This switch to IPv6 will fail under such circumstances, but because IPv4 host name resolution is already in the cache, the client can quickly switch back to IPv4 and proceed the transfer.
>
> And with the "Auto" mode bit turned on, some special clients can enforce "dual" queries even for IPv4 resolve mode, but this seems like a very special use case to me to be enabled by default.
>
> Thanks,
> Dmitry Karpov
>
>
> -----Original Message-----
> From: Daniel Stenberg<daniel_at_haxx.se>
> Sent: Saturday, July 9, 2022 3:14 PM
> To: Dmitry Karpov via curl-library<curl-library_at_lists.haxx.se>
> Cc: Dmitry Karpov<dkarpov_at_roku.com>
> Subject: RE: IPv6 resolution problems for IPv4 resolve mode
>
> On Thu, 7 Jul 2022, Dmitry Karpov via curl-library wrote:
>
>> So, the intent described in the statement "It is important that *all*
>> addresses versions are resolved, even if not used..." from
>> 84d2839740ca7804 will cause transfer timeouts for IPv4-only resolution
>> mode in case of bad/missing AAAA responses.
> I would rather say that it *may* cause timouts, as it certainly is not how things *should* work.
>
> Your patch looks like it might work. I'm still mostly concerned that this takes us back in behavior a little bit and that it will make someone else not like that and come here in a few montths with an issue about it.
>
> Wouldn't it be better if we add a bit to the CURLOPT_IPRESOLVE option that says something like "enforce this in name resolve too" ? Like:
>
> #define CURL_IPRESOLVE_ALSO_RESOLVE 0x10 /* name to be debated */
>
> to be used like this:
>
> curl_easy_setopt(c, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 |
> CURL_IPRESOLVE_ALSO_RESOLVE );
>
> ... which then makes the name resolve for this transfer also just ask for IPv4 and the connect only be done using IPv4?
>
> Maybe even make the DNS cache use family preference as a key for lookups so that we have separate IPv6-only and IPv4-only results in memory for the same host?
>
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
- application/pgp-signature attachment: OpenPGP digital signature