Re: possible off topic -- cookies from browser devtools/network

From: bruce via curl-users <curl-users_at_lists.haxx.se>
Date: Sat, 1 Jan 2022 20:25:20 -0500

Hi Ray.

Thanks for the reply.

I had initially thought, this is simple. I do a "curl" test, and use
the "-c cookieJar.txt" but the cookiear file was empty..

curl -L "https://bu.bncollge.com" -U 'user-agent txt" -cookie-jar a.dat

The thinking being, that this would/should have the data for the
headers. in the cookie file.

In researching browser extensions, I'm finding cookie apps that can be
run, to get cookies from URLs from "tabs", but I haven't run across
any logid/code that demonstrates how to invoke an xhr/fetch API
process to fetch a URL and to then get resulting cookies. I know the
cookies are "transferred" as I can see them in the Devtool/network
area of the browser.

thanks



On Sat, Jan 1, 2022 at 6:54 PM Ray Satiro via curl-users
<curl-users_at_lists.haxx.se> wrote:
>
> On 1/1/2022 3:29 PM, bruce via curl-users wrote:
> > Doing some tests, and wonder how I can invoke a testURLa nd get the
> > "Request" Cookies listed in the browser.
> >
> > My simple use test.
> >
> > 1) Create a test Javascript/extension
> > 2) Use xhr/Fetch API to generate the test url/content.
> > 3) Check in the browser/network/headers to examine cookies.
> >
> > All cookies are cleared from the browser cookie store, so the test
> > displays the cookie/headers from the test xhr/fetch call.
> >
> > I've been looking all over for how to get the Request cookies via code
> > from either a test app, or from curl on the command line..
> >
> > A test url would be "htps://bu.bncollege.com"
>
>
> Note browsers usually have a Copy as cURL option [1] in dev tools,
> though the "cmd" variant in Chrome has a bug [2].
>
> If you want to automate exporting a specific URL's cookies to a cookie
> file that can be read by curl I don't think I've had to do that... I
> think it would be possible though. You would have to load the URL in the
> browser and then convert the cookies from the browser to the Netscape
> cookie format [3] which can be read by curl.
>
> For example, with Firefox you could make a blank profile, load the URL
> and then grab all cookies by using Nir Sofer's MZCookiesView [4] to
> export the Firefox cookies database to the Netscape format by using mzcv
> /scookiestxt. (Note the utility is architecture specific, eg 32 bit
> version will show no cookies for 64 bit Firefox).
>
> A more complicated and powerful way would be Chrome's remote dev tools
> interface [5][6] which you can communicate with via websocket, and then
> do Network.getAllCookies [7] or more precise Network.getCookies with a
> URL to get the cookies of. Then you would have to convert the cookies
> from the json returned by devtools to the Netscape format. You could
> probably do that using jq or perl.
>
> There might be easier ways though I think if you are going to control
> the browser headless then that is out of scope for curl. You could
> technically build binary websocket requests and send them using curl I
> suppose but I haven't tried that, it seems like a lot of work.
>
>
> [1]: https://everything.curl.dev/usingcurl/copyas
> [2]: https://bugs.chromium.org/p/chromium/issues/detail?id=1242803
> [3]: https://everything.curl.dev/libcurl-http/cookies#cookie-file-format
> [4]: https://www.nirsoft.net/utils/mzcv.html
> [5]: https://developers.google.com/web/updates/2017/04/headless-chrome
> [6]: https://chromedevtools.github.io/devtools-protocol/
> [7]:
> https://chromedevtools.github.io/devtools-protocol/tot/Network/#method-getAllCookies
>
>
> --
> Unsubscribe: https://lists.haxx.se/listinfo/curl-users
> Etiquette: https://curl.haxx.se/mail/etiquette.html

-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-users
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Received on 2022-01-02