curl-users
Re: Need guidance using cURL and Apache authentication
Date: Mon, 10 May 2010 22:38:00 -0400
On Mon, May 10, 2010 at 6:17 PM, Charles <cecrume_at_gmail.com> wrote:
> Pardon me for "jumping the gun" so to speak -- I replied directly to Ralph
> late last night. My aplogies to Ralph and others if I shouldn't have done
> that.
>
> Anyway -- here's my current status on this
>
>
> I appreciate the reply, but don't understand what you are trying to tell
> me. I got out my cURL docs (the ones I chased down and printed off the web a
> year or so ago).
>
> I do not understand your combination of options. -o writes the data cURL
> retrieves into a file (I have used this in other cURL scripts and then
> manipulate the data via PHP).
>
> I also think I understand -L -- it is supposed to instruct cURL to go to
> the new page, right? But in my case it doesn't seem to be following to the
> page I want.
>
> I do not understand -w -- what is it actually doing? It seems to be in
> direct conflict with -o
>
> It's been a while since I've used UNIX (used to be a systems administrator
> -- these days I run Windows NT servers). I remember that the grave accents
> cause the shell to execute what's in them -- the cURL command here -- but
> just what is being stored in EFFURL? I mean... if you use the -o parameter
> why do you need EFFRUL -- isn't the output already stored in "file.html"?
>
The -o option directs the retrieved html into a file. The -w
'%url_effective' option tells curl to output the url of the page to stdout.
See the difference? "Page content" as opposed to "location of the
page". You'll need that "effective url" to convert any relative links in the
html into urls you can fetch.
As for -L, that tells curl to follow redirects in the form of Location
headers. It can't follow redirects caused by:
- meta tag with refresh to some other url
- html redirect via "window.location=<some-new-url> or similar
- javascript doing several forms of redirect, including:
- onLoad="form.submit()" in the <body> tag
- <script>form.submit()</script> at the end of the page
I once fought with a web server <cough>cognos<cough> that did all of the
above in a succession of 5 or 6 pages before I even reached the content page
I was looking for... I couldn't skip any of the intermediate pages because
they each passed their own special cookie.
BTW, I'm sure someone will mention that "top-posting" is naughty... :)
Ralph Mitchell
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2010-05-11