I brought this subject to the httpbis group once, didn't help much. It is messy.

While there hasn't been a lot of encouragement in that thread, there's also very little in the way of reports of real damage from handling absolute domains in URLs/Host request header fields. Given the massive number of requests are being done by the major browsers, if merely supporting absolute URLs caused problems, they would probably either have been fixed by now, or the browser makers would have changed the behaviour of their software. I've certainly never seen anyone complaining about wget handling it that way either.

As I see it, there are really four cases here:

• webservers that use the absolute version of their domain name as the canonical URL, and redirect requests with the non-absolute version of their domain name to the absolute version, like pyropus.ca./ above. I've seen enough other examples of this over the years, though it's pretty much impossible to craft a search query to find them. Some sites like them because it speeds up the initial request handling, as the client's resolver skips all the search-domain prefix versions it would normally try before getting around to trying it against the root.
• webservers that handle either the absolute or non-absolute version of their domain name, and serve either without redirecting. https://cbc.ca./ , https://gizmodo.com./ , https://jalopnik.com./ , even https://devblogs.microsoft.com./ ) Some will generate links using whatever domain name the original request used, others will always generate either non-absolute or absolute.
• webservers that use the non-absolute version of their domain name as the canonical URL, redirecting requests which specify the absolute version in the Host: header.
• webservers that only consider the non-absolute version of their domain name, and break in various ways (from 404 to certificate errors or other problems) if you send the absolute domain name in the Host: header.

The first three all work just fine if the user agent preserves the trailing dot in the Host header (either from a redirect, like with pyropus.ca., or from a user typing it in themselves, or from a link from elsewhere on the net). Only the fourth case breaks, and those servers never generate a redirect Host: header or link using the absolute version, so the only people that would get bitten in this case would be people who manually add a trailing dot to a link or domain name -- and pretty much nobody, to within rounding error, does that.

So given that there's no real problems being prevented by not supporting/preserving absolute hostnames in Host: headers, and that supporting them does actually fix a problem (as well as making the library's behaviour closer to the most common user-agents), it seems to me it would be worth making the change.

It looks like this used to work in curl and changed at some point.

$ /usr/bin/curl --version
curl 7.30.0 (x86_64-apple-darwin13.0) libcurl/7.30.0 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IPv6 Largefile NTLM NTLM_WB SSL libz
$ /usr/bin/curl -IL http://pyropus.ca/ --max-redirs 3
HTTP/1.1 301 Moved Permanently
Date: Tue, 18 Jan 2022 02:34:37 GMT
Server: Apache/2.4.52 (Debian)
Location: https://pyropus.ca./
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 200 OK
Date: Tue, 18 Jan 2022 02:34:37 GMT
Server: Apache/2.4.52 (Debian)
Accept-Ranges: bytes
Vary: Accept-Encoding
X-Frame-Options: sameorigin
Content-Type: text/html
$

$ /usr/bin/curl --version
curl 7.43.0 (x86_64-apple-darwin14.0) libcurl/7.43.0 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets
$ /usr/bin/curl -IL http://pyropus.ca/ --max-redirs 3
HTTP/1.1 301 Moved Permanently
Date: Tue, 18 Jan 2022 02:35:10 GMT
Server: Apache/2.4.52 (Debian)
Location: https://pyropus.ca./
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 301 Moved Permanently
Date: Tue, 18 Jan 2022 02:35:11 GMT
Server: Apache/2.4.52 (Debian)
Location: https://pyropus.ca./
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 301 Moved Permanently
Date: Tue, 18 Jan 2022 02:35:11 GMT
Server: Apache/2.4.52 (Debian)
Location: https://pyropus.ca./
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 301 Moved Permanently
Date: Tue, 18 Jan 2022 02:35:11 GMT
Server: Apache/2.4.52 (Debian)
Location: https://pyropus.ca./
Content-Type: text/html; charset=iso-8859-1

curl: (47) Maximum (3) redirects followed
$

there are really four cases here:

I think you're forgetting cases though by looking at this as HTTP only.

For example how SNI says there should be no trailing dot in there, so you'd need to send different names in the separate fields (to comply) or not, and there will be servers assuming one or the other way,

But yes, I presume we should make curl work closer to what the browsers do with the trailing dots, since they are the main clients that people are generally adapting to. How do they populate the SNI field with trailing dots?

The browsers leave the dot in the Host: header and strip it from the SNI field as that spec requires. This is what seems to work well for them. It seems like a reasonable approach, and as Ryan mentions above, curl may even have been doing this previously?

Oh, and as far as other protocols go - this two-values-are-different issue can't come up unless the agent has to send the SNI field and also send the hostname-with-a-dot in a request header field or similar. I don't think that would apply to ftp or gopher or some of the other protocols curl supports - haven't thought about it long enough to eliminate all of them.

curl may even have been doing this previously?

... and we if we did, we changed because it caused a problem in the past...

Since we need to drop the dot from the SNI, a HTTPS server cannot differentiate between a host name with a trailing dot or not. That makes you question how HTTP and HTTPS can be made that different...

Update: I think we've even had issues with HTTPS servers not liking the trailing dot when the SNI (virtual host) is setup to not use a dot.

I'm not an expert, but I believe SNI has to have the dot stripped - but the various domains listed above seem to work fine with and without it in the Host: header.

The only problems I've been able to find have been by me appending a dot to the domain name manually when typing in a URL with some webservers that can't handle it correctly (no domain configured errors etc). But those domains do not generate links or redirect URLs containing the trailing dot, so you can't run into that problem "normally" - i.e. you have to stick your hand in the saw blade to get the negative result.

Maybe some HTTPS servers don't like absolute host names in the Host: request header - but again, they won't be generating such URLs so you can only run into problems by adding it yourself. Plenty of other sites handle it just fine.

changed at some point

5de8d84

Okay, that commit message seems to be saying they needed to remove the trailing dot from the SNI value (which is true) but they also did the same for the Host: HTTP request header without including a reason for that. I think they just didn't consider the breakage doing that would cause.

The SNI change is correct, the Host: header field one is not. Can that part be reverted so curl users can execute requests with absolute domain names again?

Can that part be reverted

it is just source code, everything is possible. I doubt the commit in question will agree to "just be reverted" though after all this time and some later follow-up tweaks, so it needs some proper massaging and most likely also a test case or two added to verify.

Thank you!