Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cookies aren't saved for hostnames #3649

Closed
jay opened this issue Mar 6, 2019 · 17 comments
Closed

Cookies aren't saved for hostnames #3649

jay opened this issue Mar 6, 2019 · 17 comments
Labels
cookies HTTP regression

Comments

@jay
Copy link
Member

jay commented Mar 6, 2019

I did this

Reported by @deepakhj in the offending PR and Pierre Brico in curl-users (the latter version below):

My issue is related to the cookies usage. I've written a script which 
emulates a simple user who logs into a web site and execute a certain 
action. With curl version 7.40, everything was working as expected... With 
the new version of curl, no cookie is sent when executing the POST. 
I've searched the documentation but found no changes related to the cookie 
engine. That's the reason why I've post my issue to this mailing list...

> POST /auth/realms/fhome/login-actions/authenticate?session_code=anEjPMdR-9_Xl0t68_vDaT5yu6JoXAizauJ3rtrgcPw&execution=f65ccedb-cc7a-4f46-b386-f59408012698&client_id=XXXXXXXXXXXXX&tab_id=SwFAuhzueT0 HTTP/2 
> Host: fronteo-keycloak-rmm:8443 
> User-Agent: curl/7.64.0 
> Accept: */* 
> Content-Length: 27 
> Content-Type: application/x-www-form-urlencoded 
> 

And this output of the old version: 

> POST /auth/realms/fhome/login-actions/authenticate?session_code=yRyjuGxKaOLa-628N1ZLQi4l39Yc1g7BcBa1EwTgJvo&execution=f65ccedb-cc7a-4f46-b386-f59408012698&client_id=XXXXXXXXXXXXX&tab_id=episi_krhDY HTTP/1.1 
> User-Agent: curl/7.40.0 
> Host: fronteo-keycloak-rmm:8443 
> Accept: */* 
> Cookie: AUTH_SESSION_ID=bd7a2b29-f09d-4950-adb4-733c2d2f8cfb.fronteo-keycloak-rmm; KC_RESTART=eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyOTNiZDllMy0wMDJlLTQ4OTYtYjg0Ny0xODdlMDkzNjdmZGQifQ.eyJjaWQiOiIzNDY2Njg3YS03ZTNmLTRjOGYtYjRhMy1iZjE1ZmIxMWFhYTQiLCJwdHkiOiJvcGVuaWQtY29ubmVjdCIsInJ1cmkiOiJodHRwOi8vbG9jYWxob3N0OjgwODAiLCJhY3QiOiJBVVRIRU5USUNBVEUiLCJub3RlcyI6eyJzY29wZSI6ImFpc3AiLCJpc3MiOiJodHRwczovL2Zyb250ZW8ta2V5Y2xvYWstcm1tOjg0NDMvYXV0aC9yZWFsbXMvZmhvbWUiLCJyZXNwb25zZV90eXBlIjoiY29kZSIsImNvZGVfY2hhbGxlbmdlX21ldGhvZCI6InBsYWluIiwicmVkaXJlY3RfdXJpIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwIiwic3RhdGUiOiIxMjM0NSJ9fQ.JhMJmSKLESr_flL3SRZgmaFw4T1e5ueg-1z9OEFoRX8 
> Content-Length: 27 
> Content-Type: application/x-www-form-urlencoded 
>

Note the host fronteo-keycloak-rmm is not a DNS name so what he's seeing is likely due to #2964 which "extended domain checks to non psl builds", namely such a hostname will be unable to set cookies.

I expected the following

Non-DNS cookies to be saved... maybe. I'm not sure if this is right or not.

curl/libcurl version

curl 7.64.0 (x86_64-pc-msys) libcurl/7.64.0 OpenSSL/1.1.1b zlib/1.2.11 brotli/1.0.7 libidn2/2.1.1 libpsl/0.20.2 (+libidn2/2.1.1) libssh2/1.8.0 nghttp2/1.36.0
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS Debug TrackMemory IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz brotli TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL Metalink

operating system

Windows

/cc @danielgustafsson
@bagder
Copy link
Member

bagder commented Mar 6, 2019 •
edited

likely due to #2964

For non-PSL builds yes, as PSL builds already had that logic. Both cases now have stricter host name requirements than curl did back in the 7.40 days.

I'm not sure if this is right or not.

This is a lovely in-between case. The cookie specs are clearly moving towards making this sort of cookie illegal, as they tend to focus on how cookies (should) work on the big Internet. In this case we're talking cookies in a more closed environment.

It's a tricky situation. I would love to be able to fix this without adding another option, but can we?

@jay
Copy link
Member Author

jay commented Mar 7, 2019

For non-PSL builds yes, as PSL builds already had that logic. Both cases now have stricter host name requirements than curl did back in the 7.40 days.

Firefox and Chrome allow it. Is there a disadvantage of allowing hostname regardless of libpsl other than (I assume) some rfc violation?

/cc @rockdaboot

@bagder
Copy link
Member

bagder commented Mar 7, 2019

Firefox and Chrome allow it

You mean they allow this dot-less name? Then we really should too!

Is there a disadvantage of allowing hostname regardless of libpsl other than (I assume) some rfc violation?

Yes, it's a security thing. If "example.co.uk" can set a cookie for "co.uk", then that cookie will then also match "secrets.co.uk" etc. "co.uk" being a PSL. The RFC on this is actually really vague and PSL as a concept is weird and fragile, but that's the best there is right now...

@rockdaboot
Copy link
Contributor

On a dot-less domain (in this case it is a hostname), the "prevailing star rule" of the PSL is applied. In the internet you normally want it for cookie checking, else you open up a security hole.

But on local networks, you might not want it. So you can use psl_is_public_suffix2() with the PSL_TYPE_NO_STAR_RULE as third argument (see docs for details).

The same for the psl utility when using --no-star-rule.

I can currently not say if there is a way to automatically detect the environment. If there is no safe way, it smells for another curl option.

@danielgustafsson
Copy link
Member

I can currently not say if there is a way to automatically detect the environment. If there is no safe way, it smells for another curl option.

It seems rather complicated to detect that in way that it will a) catch all the cases and b) be leakproof wrt cookie jars.

If we add an option to this, then we need to make the option apply to PSL enabled builds too I would assume.

(sorry for being slow to respond, travelling put me offline for a bit)

@rockdaboot
Copy link
Contributor

then we need to make the option apply to PSL enabled builds too

That makes sense. Let me know how you name it (if it comes to that point). I would like to see something similar for wget/wget2. Can't hurt if the option names match.

@rockdaboot
Copy link
Contributor

If you use curl with libpsl, you could also add your local domain/host name as exception in the used PSL file.

@bagder
Copy link
Member

bagder commented Mar 7, 2019

We should support this without any extra options since the browsers do that. I can go to http://moo/cookie with Firefox and it can set cookies - for that host name.

@bagder
Copy link
Member

bagder commented Mar 8, 2019

I noticed the bug is only present in non-PSL builds. I think I'll go with reverting 3773de3. PR coming up.

bagder added a commit that referenced this issue Mar 8, 2019
@bagder
test331: verify set-cookie for dot-less host name
912caf1 
Reproduced bug #3649
@bagder bagder mentioned this issue Mar 8, 2019
Closed
@bagder
Copy link
Member

bagder commented Mar 8, 2019

This bug was introduced in 7.64.0.

@jay
Copy link
Member Author

jay commented Mar 8, 2019

Are you sure this is only non-psl? The guy who reported it seems to be using libpsl/0.20.2

@bagder
Copy link
Member

bagder commented Mar 8, 2019

My new test case in #3659 worked fine with my PSL build when I tried it locally, and the commit I reverted to get it working on a non-PSL build was only affecting non-PSL.

If the problem exists on a PSL build, then I suspect my test case isn't good enough. Unfortunately I don't think we have the full URL and its corresponding response headers to know for sure?

@bagder bagder closed this as completed in 299d966 Mar 10, 2019
bagder added a commit that referenced this issue Mar 10, 2019
@bagder
test331: verify set-cookie for dotless host name
ff7e5a2 
Reproduced bug #3649
Closes #3659
@Largo2005
Copy link

For you information, I've noticed the problem using the latest version of MSYS2 which embeds curl 7.64.0. It seems that this curl was built with PSL feature as mentioned by the --version option:
$ curl --version
curl 7.64.0 (x86_64-pc-msys) libcurl/7.64.0 OpenSSL/1.1.1b zlib/1.2.11 brotli/1.0.7 libidn2/2.1.1 libpsl/0.20.2 (+libidn2/2.1.1) libssh2/1.8.0 nghttp2/1.36.0
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS Debug TrackMemory IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz brotli TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL Metalink

@bagder
Copy link
Member

bagder commented Mar 11, 2019 •
edited

I've noticed the problem

Then please detail the problem for us. The problem as I reproduced (and fixed) was not present in PSL builds. How do you reproduce it?

@bagder
Copy link
Member

bagder commented Mar 12, 2019

Primarily, I would like to know the URL used and what the (cookie) response headers looked like that curl didn't accept.

@Largo2005
Copy link

Largo2005 commented Mar 12, 2019 •
edited by jay

OK, as I already told, I use a binary version of curl embedded in MSYS2 (freely downloadable at https://www.msys2.org/). This binary is built with the features mentioned above.
My problem is linked to local hostname. The URL I would like curl to connect to is "https://fronteo-keycloak-rmm:8443" (a Redhat SSO server instance). Here is a log (I've removed some useless lines):

> GET /auth/realms/fhome/protocol/openid-connect/auth?response_type=code&scope=aisp&client_id=3466687a-7e3f-4c8f-b4a3-bf15fb11aaa4&state=12345&redirect_uri=http://localhost:8080 HTTP/2
> Host: fronteo-keycloak-rmm:8443
> User-Agent: curl/7.64.0
> Accept: */*
>
* STATE: DO => DO_DONE handle 0x600055128; line 1716 (connection #0)
* multi changed, check CONNECT_PEND queue!
* STATE: DO_DONE => WAITPERFORM handle 0x600055128; line 1840 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x600055128; line 1855 (connection #0)
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
} [5 bytes data]
* multi changed, check CONNECT_PEND queue!
{ [5 bytes data]
* HTTP/2 found, allow multiplexing
< HTTP/2 200
< cache-control: no-store, must-revalidate, max-age=0
* Added cookie AUTH_SESSION_ID="8696e8d7-3b93-4687-a9a4-12dc9ece0cb4.fronteo-keycloak-rmm" for domain fronteo-keycloak-rmm, path /auth/realms/fhome/, expire 0
< set-cookie: AUTH_SESSION_ID=8696e8d7-3b93-4687-a9a4-12dc9ece0cb4.fronteo-keycloak-rmm; Version=1; Path=/auth/realms/fhome/; HttpOnly
* Added cookie KC_RESTART="eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyOTNiZDllMy0wMDJlLTQ4OTYtYjg0Ny0xODdlMDkzNjdmZGQifQ.eyJjaWQiOiIzNDY2Njg3YS03ZTNmLTRjOGYtYjRhMy1iZjE1ZmIxMWFhYTQiLCJwdHkiOiJvcGVuaWQtY29ubmVjdCIsInJ1cmkiOiJodHRwOi8vbG9jYWxob3N0OjgwODAiLCJhY3QiOiJBVVRIRU5USUNBVEUiLCJub3RlcyI6eyJzY29wZSI6ImFpc3AiLCJpc3MiOiJodHRwczovL2Zyb250ZW8ta2V5Y2xvYWstcm1tOjg0NDMvYXV0aC9yZWFsbXMvZmhvbWUiLCJyZXNwb25zZV90eXBlIjoiY29kZSIsImNvZGVfY2hhbGxlbmdlX21ldGhvZCI6InBsYWluIiwicmVkaXJlY3RfdXJpIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwIiwic3RhdGUiOiIxMjM0NSJ9fQ.JhMJmSKLESr_flL3SRZgmaFw4T1e5ueg-1z9OEFoRX8" for domain fronteo-keycloak-rmm, path /auth/realms/fhome/, expire 0
< set-cookie: KC_RESTART=eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyOTNiZDllMy0wMDJlLTQ4OTYtYjg0Ny0xODdlMDkzNjdmZGQifQ.eyJjaWQiOiIzNDY2Njg3YS03ZTNmLTRjOGYtYjRhMy1iZjE1ZmIxMWFhYTQiLCJwdHkiOiJvcGVuaWQtY29ubmVjdCIsInJ1cmkiOiJodHRwOi8vbG9jYWxob3N0OjgwODAiLCJhY3QiOiJBVVRIRU5USUNBVEUiLCJub3RlcyI6eyJzY29wZSI6ImFpc3AiLCJpc3MiOiJodHRwczovL2Zyb250ZW8ta2V5Y2xvYWstcm1tOjg0NDMvYXV0aC9yZWFsbXMvZmhvbWUiLCJyZXNwb25zZV90eXBlIjoiY29kZSIsImNvZGVfY2hhbGxlbmdlX21ldGhvZCI6InBsYWluIiwicmVkaXJlY3RfdXJpIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwIiwic3RhdGUiOiIxMjM0NSJ9fQ.JhMJmSKLESr_flL3SRZgmaFw4T1e5ueg-1z9OEFoRX8; Version=1; Path=/auth/realms/fhome/; HttpOnly
< x-xss-protection: 1; mode=block
< x-frame-options: SAMEORIGIN
< content-security-policy: frame-src 'self'; frame-ancestors 'self'; object-src 'none';
< date: Tue, 12 Mar 2019 14:22:06 GMT
< x-robots-tag: none
< strict-transport-security: max-age=31536000; includeSubDomains
< x-content-type-options: nosniff
< content-type: text/html;charset=utf-8
< content-language: fr
<
{ [15509 bytes data]
* nread <= 0, server closed connection, bailing
* STATE: PERFORM => DONE handle 0x600055128; line 2052 (connection #0)
* multi_done
* Connection #0 to host fronteo-keycloak-rmm left intact
* Expire cleared (transfer 0x600055128)
POST URL https://fronteo-keycloak-rmm:8443/auth/realms/fhome/login-actions/authenticate?session_code=GJrBBqxLoIhXdOGItEhE3qPOFX380SelqRveHwaYBxA&execution=f65ccedb-cc7a-4f46-b386-f59408012698&client_id=3466687a-7e3f-4c8f-b4a3-bf15fb11aaa4&tab_id=Y4joE-fW6mE
* Expire in 0 ms for 6 (transfer 0x600055128)
* STATE: INIT => CONNECT handle 0x600055128; line 1443 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* Expire in 1 ms for 1 (transfer 0x600055128)
* STATE: CONNECT => WAITRESOLVE handle 0x600055128; line 1484 (connection #0)
* Expire in 0 ms for 1 (transfer 0x600055128)
* Expire in 1 ms for 1 (transfer 0x600055128)
*   Trying 192.100.2.31...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x600055128)
* STATE: WAITRESOLVE => WAITCONNECT handle 0x600055128; line 1565 (connection #0)
* Connected to fronteo-keycloak-rmm (192.100.2.31) port 8443 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x600055128; line 1619 (connection #0)
* Marked for [keep alive]: HTTP default
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /usr/ssl/certs/ca-bundle.crt
  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x600055128; line 1633 (connection #0)
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [94 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [815 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=BE; L=LLN; O=Mainsys; CN=keycloak
*  start date: Feb 11 14:23:59 2019 GMT
*  expire date: Feb 11 14:23:59 2039 GMT
*  issuer: C=BE; L=LLN; O=Mainsys; CN=keycloak
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* STATE: PROTOCONNECT => DO handle 0x600055128; line 1654 (connection #0)
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x600055128)
} [5 bytes data]
> POST /auth/realms/fhome/login-actions/authenticate?session_code=GJrBBqxLoIhXdOGItEhE3qPOFX380SelqRveHwaYBxA&execution=f65ccedb-cc7a-4f46-b386-f59408012698&client_id=3466687a-7e3f-4c8f-b4a3-bf15fb11aaa4&tab_id=Y4joE-fW6mE HTTP/2
> Host: fronteo-keycloak-rmm:8443
> User-Agent: curl/7.64.0
> Accept: */*
> Content-Length: 27
> Content-Type: application/x-www-form-urlencoded
>
* STATE: DO => DO_DONE handle 0x600055128; line 1716 (connection #0)
* multi changed, check CONNECT_PEND queue!
* STATE: DO_DONE => WAITPERFORM handle 0x600055128; line 1840 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x600055128; line 1855 (connection #0)
} [5 bytes data]
* We are completely uploaded and fine
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
} [5 bytes data]
* multi changed, check CONNECT_PEND queue!
{ [5 bytes data]
* HTTP/2 found, allow multiplexing
< HTTP/2 500
< cache-control: no-store, must-revalidate, max-age=0
< content-length: 0
< date: Tue, 12 Mar 2019 14:22:06 GMT
<

Here is the file containing the cookies:
cookies-in.txt

Hope this helps,
Pierre

@bagder bagder mentioned this issue Mar 12, 2019
Closed
@bagder
Copy link
Member

bagder commented Mar 12, 2019

I turned that into a "real" issue: #3676 so that it gets the correct attention.

@lock lock bot locked as resolved and limited conversation to collaborators Jun 10, 2019
@bagder bagder added the cookies label Nov 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
cookies HTTP regression
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cookie: handle dotless host name in non-PSL build curl/curl
5 participants
@bagder @danielgustafsson @jay @rockdaboot @Largo2005