Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: possible off topic -- cookies from browser devtools/network
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Ray Satiro via curl-users <curl-users_at_lists.haxx.se>
Date: Sat, 1 Jan 2022 18:54:07 -0500
On 1/1/2022 3:29 PM, bruce via curl-users wrote:
> Doing some tests, and wonder how I can invoke a testURLa nd get the
> "Request" Cookies listed in the browser.
>
> My simple use test.
>
> 1) Create a test Javascript/extension
> 2) Use xhr/Fetch API to generate the test url/content.
> 3) Check in the browser/network/headers to examine cookies.
>
> All cookies are cleared from the browser cookie store, so the test
> displays the cookie/headers from the test xhr/fetch call.
>
> I've been looking all over for how to get the Request cookies via code
> from either a test app, or from curl on the command line..
>
> A test url would be "htps://bu.bncollege.com"
Note browsers usually have a Copy as cURL option [1] in dev tools,
though the "cmd" variant in Chrome has a bug [2].
If you want to automate exporting a specific URL's cookies to a cookie
file that can be read by curl I don't think I've had to do that... I
think it would be possible though. You would have to load the URL in the
browser and then convert the cookies from the browser to the Netscape
cookie format [3] which can be read by curl.
For example, with Firefox you could make a blank profile, load the URL
and then grab all cookies by using Nir Sofer's MZCookiesView [4] to
export the Firefox cookies database to the Netscape format by using mzcv
/scookiestxt. (Note the utility is architecture specific, eg 32 bit
version will show no cookies for 64 bit Firefox).
A more complicated and powerful way would be Chrome's remote dev tools
interface [5][6] which you can communicate with via websocket, and then
do Network.getAllCookies [7] or more precise Network.getCookies with a
URL to get the cookies of. Then you would have to convert the cookies
from the json returned by devtools to the Netscape format. You could
probably do that using jq or perl.
There might be easier ways though I think if you are going to control
the browser headless then that is out of scope for curl. You could
technically build binary websocket requests and send them using curl I
suppose but I haven't tried that, it seems like a lot of work.
[1]: https://everything.curl.dev/usingcurl/copyas
[2]: https://bugs.chromium.org/p/chromium/issues/detail?id=1242803
[3]: https://everything.curl.dev/libcurl-http/cookies#cookie-file-format
[4]: https://www.nirsoft.net/utils/mzcv.html
[5]: https://developers.google.com/web/updates/2017/04/headless-chrome
[6]: https://chromedevtools.github.io/devtools-protocol/
[7]:
https://chromedevtools.github.io/devtools-protocol/tot/Network/#method-getAllCookies
Date: Sat, 1 Jan 2022 18:54:07 -0500
On 1/1/2022 3:29 PM, bruce via curl-users wrote:
> Doing some tests, and wonder how I can invoke a testURLa nd get the
> "Request" Cookies listed in the browser.
>
> My simple use test.
>
> 1) Create a test Javascript/extension
> 2) Use xhr/Fetch API to generate the test url/content.
> 3) Check in the browser/network/headers to examine cookies.
>
> All cookies are cleared from the browser cookie store, so the test
> displays the cookie/headers from the test xhr/fetch call.
>
> I've been looking all over for how to get the Request cookies via code
> from either a test app, or from curl on the command line..
>
> A test url would be "htps://bu.bncollege.com"
Note browsers usually have a Copy as cURL option [1] in dev tools,
though the "cmd" variant in Chrome has a bug [2].
If you want to automate exporting a specific URL's cookies to a cookie
file that can be read by curl I don't think I've had to do that... I
think it would be possible though. You would have to load the URL in the
browser and then convert the cookies from the browser to the Netscape
cookie format [3] which can be read by curl.
For example, with Firefox you could make a blank profile, load the URL
and then grab all cookies by using Nir Sofer's MZCookiesView [4] to
export the Firefox cookies database to the Netscape format by using mzcv
/scookiestxt. (Note the utility is architecture specific, eg 32 bit
version will show no cookies for 64 bit Firefox).
A more complicated and powerful way would be Chrome's remote dev tools
interface [5][6] which you can communicate with via websocket, and then
do Network.getAllCookies [7] or more precise Network.getCookies with a
URL to get the cookies of. Then you would have to convert the cookies
from the json returned by devtools to the Netscape format. You could
probably do that using jq or perl.
There might be easier ways though I think if you are going to control
the browser headless then that is out of scope for curl. You could
technically build binary websocket requests and send them using curl I
suppose but I haven't tried that, it seems like a lot of work.
[1]: https://everything.curl.dev/usingcurl/copyas
[2]: https://bugs.chromium.org/p/chromium/issues/detail?id=1242803
[3]: https://everything.curl.dev/libcurl-http/cookies#cookie-file-format
[4]: https://www.nirsoft.net/utils/mzcv.html
[5]: https://developers.google.com/web/updates/2017/04/headless-chrome
[6]: https://chromedevtools.github.io/devtools-protocol/
[7]:
https://chromedevtools.github.io/devtools-protocol/tot/Network/#method-getAllCookies
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-users Etiquette: https://curl.haxx.se/mail/etiquette.htmlReceived on 2022-01-02