Aha!! (Thanks Ralph!!)

My apologies folks.. I had tested/retested on a couple of browsers with
cookies removed/javascript disabled to test.. When I saw it worked, went
with the process of curl working as javascript turned off still generated
cookies in the browser..

My bad.. for some reason, turnss out hadn't really deleted cookies. When I
rechecked/tested with cookies off/javascript really disabled -- no cookies
in the brower!

Which then led to me doing a quick test using casperjs to perform my test.

The sample casperjs chunk of code follows. It does the process for the
url1/url2 while handling the required cookies internally. All I'm really
interested in is the generation of the url which is passed in..

/*
*
* mstate2.js: test to take input url
*
* casper.js mstate2.js --url="foo"
*
*
*/
var casper = require('casper').create({
    verbose: true,
    logLevel: "debug",

    pageSettings:
        {
            userAgent: 'Mozilla/5.0 (X11; Linux x86_64; rv:24.0)
Gecko/20100101 Firefox/24.0'
        }
});

phantom.timeout=50000;
phantom.stepTimeout=50000;

casper.options.timeout = 200000;

var x = require('casper').selectXPath;
phantom.cookiesEnabled = true;

var url = casper.cli.get("url");

url_start='http://www.bkstr.com'

casper.start(url_start, function() {

    casper.waitForSelector(x("//a[@id='_footer_link']"),
        function success() {
            this.echo('');
         },
        function fail() {
            this.echo('');
        }
    );

    casper.thenOpen(url, function() {
        this.echo(this.page.content);
    });
});

//casper.start(url_start);

casper.run(function() {
    this.exit();
});

On Thu, May 5, 2016 at 10:13 PM, Ralph Mitchell <ralphmitchell_at_gmail.com>
wrote:

> On Thu, May 5, 2016 at 6:55 PM, bruce <badouglas_at_gmail.com> wrote:
>
>> Hi Thanks for the eyeballs on my typos..
>>
>> However, running the curls.. still don't get cookies..
>>
>> Any thoughts?
>>
>> As I stated and the weird thing, I can clear cookies in FF, and use the
>> 1st url, and generate cookies.. As far as I can tell, I'm replicating the
>> traffic/data I see from livehttpheaders over the wire...
>>
>>
>> echo "" > aa.lwp
>> curl -vvv -A 'Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
>> Firefox/38.0' -H 'Accept:
>> text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H
>> 'Accept-Language: en-US,en;q=0.5' -H 'Connection: keep-alive'
>> --cookie-jar aa.lwp --cookie aa.lwp -L '
>> http://www.bkstr.com/missouristatestore/home'
>>
>
>
> Did you look at the page being returned from the above curl call? It
> contains an enormous amount of scripting, including this little gem:
>
> dojo.addOnLoad(function() {
> SearchJS.init();
> readAkamaiCookieValues();
> enableDisableMiniCartLink();
> });
>
>
> Curl doesn't attempt to process the scripting at all, it just hands you
> the raw page. A browser processes the page as it goes along, executing
> javascript and fetching page elements/style sheets/whatever as it renders
> the page. I wouldn't be surprised to find one of the javascript files
> generates cookies on the fly. Perhaps not by the readAkamaiCookieValues()
> function, but by some script fragment.
>
> Ralph Mitchell
>
>
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
> <#m_8446801294831799101_DDB4FAA8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> -------------------------------------------------------------------
> List admin: https://cool.haxx.se/list/listinfo/curl-users
> FAQ: https://curl.haxx.se/docs/faq.html
> Etiquette: https://curl.haxx.se/mail/etiquette.html
>
>

-------------------------------------------------------------------
List admin: https://cool.haxx.se/list/listinfo/curl-users
FAQ: https://curl.haxx.se/docs/faq.html
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2016-05-06