On 9/26/2016 9:34 PM, Sergei Kuzmin wrote:
> I recently tried to automate login into av.by <http://av.by>. After
> login site sets .av.by <http://av.by> cookies (note leading dot).
> However these cookies are stored as av.by <http://av.by> (strict
> domain) so it doesn't work for sub-domain in consecutive call. Is it
> some security measure? Is there some well known workaround? At present
> I postfix the cookiejar after the call:
> sed -i -r 's/^av.by <http://av.by>\tFALSE/.av.by
> <http://av.by>\tTRUE/' av_by.cookiejar
>
> Interesting parts are in yellow:
>
> curl http://av.by/login -H 'Origin: http://av.by' -H 'Content-Type:
> application/x-www-form-urlencoded' -H 'Cache-Control: max-age=0' -H
> 'Referer: http://av.by/login' --data
> 'login_form_csrf=111111111111111111-1111111111111111111111111111&login=11111&password=1111111111&submit=%D0%92%D0%BE%D0%B9%D1%82%D0%B8&remember=0&remember=1'
> -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language:
> en-US,en;q=0.8,ru;q=0.6,pl;q=0.4' -H 'Upgrade-Insecure-Requests: 1' -H
> 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116
> Safari/537.36' -H 'Accept:
> text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
> -H 'Connection: keep-alive' --compressed -b av_by.cookiejar -c
> av_by.cookiejar -v
>
> * Connected to av.by <http://av.by> (86.57.246.106) port 80 (#0)
> > POST /login HTTP/1.1
> > Host: av.by <http://av.by>
> > Cookie: PHPSESSID=22222222222222222222222
> > Origin: http://av.by
> > Content-Type: application/x-www-form-urlencoded
> > Cache-Control: max-age=0
> > Referer: http://av.by/login
> > Accept-Encoding: gzip, deflate, sdch
> > Accept-Language: en-US,en;q=0.8,ru;q=0.6,pl;q=0.4
> > Upgrade-Insecure-Requests: 1
> > User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36
> > Accept:
> text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
> > Connection: keep-alive
> > Content-Length: 111
> >
> } [111 bytes data]
> * upload completely sent off: 111 out of 111 bytes
> < HTTP/1.1 302 Found
> < Server: nginx
> < Content-Type: text/html
> < Transfer-Encoding: chunked
> < Connection: keep-alive
> < Expires: Thu, 1 Nov 1981 01:01:01 GMT
> < Cache-Control: no-store, no-cache, must-revalidate, post-check=0,
> pre-check=0
> < Pragma: no-cache
> * Added cookie avby_id="11111" for domain av.by <http://av.by>, path
> /, expire 11111111111
> < Set-Cookie: avby_id=11111; expires=Mon, 1-Jan-2026 01:01:01 GMT;
> path=/; domain=.av.by <http://av.by>
> * Added cookie avby_hash="123456789123456789123456789"for domain av.by
> <http://av.by>, path /, expire 1111111111
> < Set-Cookie: avby_hash=123456789123456789123456789; expires=Mon,
> 1-Jan-2026 01:01:01 GMT; path=/; domain=.av.by <http://av.by>
> * Replaced cookie avby_id="11111" for domain av.by <http://av.by>,
> path /, expire 123512345 Set-Cookie: avby_id=11111; expires=Mon,
> 1-Jan-2026 01:01:01; path=/
> * Replaced cookie avby_hash="123456789123456789123456789" for domain
> av.by <http://av.by>, path /, expire 1790298555
> < Set-Cookie: avby_hash=123456789123456789123456789; expires=Mon,
> 1-Jan-2026 01:01:01; GMT; path=/
> * Replaced cookie PHPSESSID="deleted" for domain av.by <http://av.by>,
> path /, expire 1
> < Set-Cookie: PHPSESSID=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT
> * Added cookie PHPSESSID="1234512345" for domain av.by <http://av.by>,
> path /, expire 0
> < Set-Cookie: PHPSESSID=1234512345
> < Location: http://av.by/profile
>
> Cookie jar after execution
> av.by <http://av.by> FALSE / FALSE 11111111111 avby_id
> 11111
> av.by <http://av.by> FALSE / FALSE 11111111111
> avby_hash 123456789123456789123456789
> av.by <http://av.by> FALSE / FALSE 0 PHPSESSID
> 1234512345
>
> First two lines are expected to start with ".av.by <http://av.by> TRUE"
>
> Then site redirects to cars.av.by <http://cars.av.by> which doesn't
> have required cookies.

What is happening is your website is sending two Set-Cookie lines for
those cookies, one with the domain (ie yes subdomains) and one without
the domain (ie no subdomains). For example avby_id, Firefox and Chrome
will keep both cookies and send both on a future request to av.by.
libcurl however will replace the first cookie with a second cookie of
the same name, even if the 'allow subdomains' flag is different.

curld -v -b cookies.txt -c cookies.txt
"http://httpbin.org/response-headers?Set-Cookie=%20foo=bar;%20domain=.httpbin.org&Set-Cookie=%20foo=qux;"

* Added cookie foo="bar" for domain httpbin.org, path /, expire 0
< Set-Cookie: foo=bar; domain=.httpbin.org
* Replaced cookie foo="qux" for domain httpbin.org, path /, expire 0
< Set-Cookie: foo=qux;

And that results in a single cookie that does not include subdomains
(second field is 'FALSE'):
httpbin.org FALSE / FALSE 0 foo qux

So is this correct behavior? I don't know. RFC 6265 says for the server
in 4.1.1 Syntax [1]:

"Servers SHOULD NOT include more than one Set-Cookie header field in the
same response with the same cookie-name. (See Section 5.2 for how user
agents handle this case.)"

and later, in 4.1.2 Semantics [2]:

"If the user agent receives a new cookie with the same cookie-name,
domain-value, and path-value as a cookie that it has already stored, the
existing cookie is evicted and replaced with the new cookie."

... and later, in 5.3 Storage Model [3] if there is no domain attribute
"Set the cookie's domain to the canonicalized request-host."

So that could be interpreted as the domain value being the same (like
Domain=foo is domain-value 'foo' and no domain is also domain-value
'foo' if the server is foo). I've noticed in the browsers though when
there's a domain they store it as '.foo' and when there's no domain they
store it as 'foo'. I must have missed that part in the RFC? But in that
case I see an argument that the domain-value wouldn't be the same. Thoughts?

Also, a side note: when Firefox or Chrome send two cookies of the same
name but one allowing subdomains how does a server know which is which?
I'm looking at RFC 6265 and I don't see any guidance for it. I did some
experiment in Chrome just now, expecting the order to be the subdomain
cookie always comes first but it doesn't, instead it's FIFO style.

[1]: https://tools.ietf.org/html/rfc6265#page-10
[2]: https://tools.ietf.org/html/rfc6265#section-4.1.2
[3]: https://tools.ietf.org/html/rfc6265#page-23

-------------------------------------------------------------------
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2016-09-27