At 05:25 PM 2/27/02 +0100, you wrote:
>On Wed, 27 Feb 2002, Yanhui Liu wrote:
>
> > I am trying to access stock quote page on msn.com site using cURL, however
> > I could not get it to work. Could you help me out? I am using curl 7.9.4
> > (i686-pc-linux-gnu) libcurl 7.9.4 (OpenSSL 0.9.6) on Redhat Linux 7.1.
>
>7.9.4 has a SSL read bug that will make it unreliable for SSL downloads. You
>should get a 7.9.5 pre-release instead, as it will work better.
>
>It would help a lot if you first of all upgraded and retried this, then come
>back if it still doesn't work.

I upgraded to 7.9.5-pre4, however I still could not get the target page.
Test results are attached at the end.

>I would also appreciate if you would be able to cut out a few issues at a
>time and ask specificly about them. It is very hard, and breath-taking to
>only parse through such a huge mail with many complex command lines involved.

Sorry about the lengthy mail, I just want to present all relevant
information for the problem. For me, I could not cut the problem into
pieces, I am totally lost.

> > 1. Is it possible for curl to use Netscape's cookie? So we can get to the
> > client state using Netscape as a tool.
>
>Yes, curl can read Netscape's cookies. Use -b for reading and -c can even
>write them back in Netscape format.

Wonderful. Does it mean curl can use netscape's cookie to retrieve pages?
For example, I ran the curl test using netscape's cookie, which was
generated after browsing the content.

$ cp ~/.netscape/cookies .
$ curl -A "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" -b cookies
-L -v -i -s -o junk.DATA --url
"http://moneycentral.msn.com/investor/quotes/pprtq.asp?Page=RTQ&Symbol=orcl"
* Connected to moneycentral.com (207.46.189.14)
> GET /investor/quotes/pprtq.asp?Page=RTQ&Symbol=orcl HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Host: moneycentral.msn.com
Pragma: no-cache
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Cookie:
QUAUTH=66504305614a51425d57755e515248404e585a5478525906280e40640357114307215e07164b43455009;
MC1=GUID=00D22A5A9CA043E6A8730ABFDF1F3195

* Follow to new URL:
/pplogin.asp?Page=http://moneycentral.msn.com/investor/quotes/pprtq.asp&Query=Page%3DRTQ%26Symbol%3Dorcl%26REQUEST%5FMETHOD%3DGET&AuthTime=43200&ForceLogin=False
* Closing connection #0
* Follows Location: to new URL:
'http://moneycentral.msn.com/pplogin.asp?Page=http://moneycentral.msn.com/investor/quotes/pprtq.asp&Query=Page%3DRTQ%26Symbol%3Dorcl%26REQUEST%5FMETHOD%3DGET&AuthTime=43200&ForceLogin=False'
* Disables POST, goes with GET
* Connected to moneycentral.com (207.46.189.14)
> GET
/pplogin.asp?Page=http://moneycentral.msn.com/investor/quotes/pprtq.asp&Query=Page%3DRTQ%26Symbol%3Dorcl%26REQUEST%5FMETHOD%3DGET&AuthTime=43200&ForceLogin=False
HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Host: moneycentral.msn.com
Pragma: no-cache
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Cookie:
QUAUTH=66504305614a51425d57755e515248404e585a5478525906280e40640357114307215e07164b43455009;
MC1=GUID=00D22A5A9CA043E6A8730ABFDF1F3195

* Follow to new URL:
http://login.passport.com/login.srf?lc=1033&id=229&ru=http%3A%2F%2Fmoneycentral%2Emsn%2Ecom%2Fpploggedin%2Easp%3FPage%3Dhttp%253A%252F%252Fmoneycentral%252Emsn%252Ecom%252Finvestor%252Fquotes%252Fpprtq%252Easp%26Query%3DPage%253DRTQ%2526Symbol%253Dorcl%2526REQUEST%255FMETHOD%253DGET&tw=43200&kv=2&ct=1014877476&ver=2.0.0248.1&tpf=5b566e78f24697f753fdd1b608fd10b3
* Closing connection #0
* Follows Location: to new URL:
'http://login.passport.com/login.srf?lc=1033&id=229&ru=http%3A%2F%2Fmoneycentral%2Emsn%2Ecom%2Fpploggedin%2Easp%3FPage%3Dhttp%253A%252F%252Fmoneycentral%252Emsn%252Ecom%252Finvestor%252Fquotes%252Fpprtq%252Easp%26Query%3DPage%253DRTQ%2526Symbol%253Dorcl%2526REQUEST%255FMETHOD%253DGET&tw=43200&kv=2&ct=1014877476&ver=2.0.0248.1&tpf=5b566e78f24697f753fdd1b608fd10b3'
* Disables POST, goes with GET
* Connected to login.passport.com (64.4.60.254)
> GET
/login.srf?lc=1033&id=229&ru=http%3A%2F%2Fmoneycentral%2Emsn%2Ecom%2Fpploggedin%2Easp%3FPage%3Dhttp%253A%252F%252Fmoneycentral%252Emsn%252Ecom%252Finvestor%252Fquotes%252Fpprtq%252Easp%26Query%3DPage%253DRTQ%2526Symbol%253Dorcl%2526REQUEST%255FMETHOD%253DGET&tw=43200&kv=2&ct=1014877476&ver=2.0.0248.1&tpf=5b566e78f24697f753fdd1b608fd10b3
HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Host: login.passport.com
Pragma: no-cache
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Cookie: MSPPre=llp_gapper_at_yahoo.com

* Closing connection #0

I still ended up at the login page. The content of the cookie file from
netscape was shown in my first posting. So curl does not pass the cookie
back to server correctly.

> > 2. It would be terrific if cURL had a recording function with simple GUI
> > browser, so you can use the GUI to access the page and curl records all
> > necessary info to get there. It will save a lot of switches and flatten the
> > learning curve to use cURL.
>
>I agree. Will you write one? ;-)

After second thought, I think if curl can use cookies from Netscape/IE,
then we don't need this recording ability unless Netscape/IE change their
format frequently. Sure, I can give a hand on this, however I am not
familiar with HTTP/SSH protocol, a lot of stuff needs to read.

Thanks.

Yanhui

Here is the failed test case with curl 7.9.5-pre4,

$ curl -A "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" -D headers
-v -i -s -d
"lc=1033&id=3&ru=&tw=20&da=passport.com&notinframe=1&login=llp_gapper_at_yahoo.com&passwd=ladder&submit1=+Sign+In+"
-o /tmp/junk --url "http://login.passport.com/login.srf"
* Connected to login.passport.com (64.4.59.254)
> POST /login.srf HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Host: login.passport.com
Pragma: no-cache
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Content-Length: 110
Content-Type: application/x-www-form-urlencoded

lc=1033&id=3&ru=&tw=20&da=passport.com&notinframe=1&login=llp_gapper_at_yahoo.com&passwd=ladder&submit1=+Sign+In+*
Closing connection #0
$
$
$ more headers
HTTP/1.1 100 Continue
Server: Microsoft-IIS/5.0
Date: Thu, 28 Feb 2002 06:27:57 GMT

HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Date: Thu, 28 Feb 2002 06:27:57 GMT
Connection: close
Expires: Thu, 28 Feb 2002 06:26:57 GMT
Cache-Control: no-cache
cachecontrol: no-store
Pragma: no-cache
P3P: CP="DSP CUR OTPi IND OTRi ONL FIN"
Content-Type: text/html
Content-Length: 18991
Set-Cookie: MSPRequ=lt=1014877677&co=1&id=3
Set-Cookie: BrowserTest=Success?; domain=.passport.com;path=/;version=1

$ curl -A "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" -b headers
-L -v -i -s -o junk.DATA --url
"http://moneycentral.msn.com/investor/quotes/pprtq.asp?Page=RTQ&Symbol=orcl"
* Connected to moneycentral.com (207.46.189.15)
> GET /investor/quotes/pprtq.asp?Page=RTQ&Symbol=orcl HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Host: moneycentral.msn.com
Pragma: no-cache
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Cookie: MSPRequ=lt=1014877677&co=1&id=3

* Follow to new URL:
/pplogin.asp?Page=http://moneycentral.msn.com/investor/quotes/pprtq.asp&Query=Page%3DRTQ%26Symbol%3Dorcl%26REQUEST%5FMETHOD%3DGET&AuthTime=43200&ForceLogin=False
* Closing connection #0
* Follows Location: to new URL: 'http://moneycentral.msn.com/pplogin.asp?Page
=http://moneycentral.msn.com/investor/quotes/pprtq.asp&Query=Page%3DRTQ%26Sym
bol%3Dorcl%26REQUEST%5FMETHOD%3DGET&AuthTime=43200&ForceLogin=False'
* Disables POST, goes with GET
* Connected to moneycentral.com (207.46.189.15)
> GET /pplogin.asp?Page=http://moneycentral.msn.com/investor/quotes/pprtq.asp
&Query=Page%3DRTQ%26Symbol%3Dorcl%26REQUEST%5FMETHOD%3DGET&AuthTime=43200&For
ceLogin=False HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Host: moneycentral.msn.com
Pragma: no-cache
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Cookie: MC1=GUID=CA5A777117C24BCAAAC38AE2E047B7F7; MSPRequ=lt=1014877677&co=1
&id=3

* Follow to new URL: http://login.passport.com/login.srf?lc=1033&id=229&ru=ht
tp%3A%2F%2Fmoneycentral%2Emsn%2Ecom%2Fpploggedin%2Easp%3FPage%3Dhttp%253A%252
F%252Fmoneycentral%252Emsn%252Ecom%252Finvestor%252Fquotes%252Fpprtq%252Easp%
26Query%3DPage%253DRTQ%2526Symbol%253Dorcl%2526REQUEST%255FMETHOD%253DGET&tw=
43200&kv=2&ct=1014877682&ver=2.0.0248.1&tpf=978415caf19b4eaa5a58f6ccd969d453
* Closing connection #0
* Follows Location: to new URL: 'http://login.passport.com/login.srf?lc=1033&
id=229&ru=http%3A%2F%2Fmoneycentral%2Emsn%2Ecom%2Fpploggedin%2Easp%3FPage%3Dh
ttp%253A%252F%252Fmoneycentral%252Emsn%252Ecom%252Finvestor%252Fquotes%252Fpp
rtq%252Easp%26Query%3DPage%253DRTQ%2526Symbol%253Dorcl%2526REQUEST%255FMETHOD
%253DGET&tw=43200&kv=2&ct=1014877682&ver=2.0.0248.1&tpf=978415caf19b4eaa5a58f
6ccd969d453'
* Disables POST, goes with GET
* Connected to login.passport.com (64.4.60.254)
> GET /login.srf?lc=1033&id=229&ru=http%3A%2F%2Fmoneycentral%2Emsn%2Ecom%2Fpp
loggedin%2Easp%3FPage%3Dhttp%253A%252F%252Fmoneycentral%252Emsn%252Ecom%252Fi
nvestor%252Fquotes%252Fpprtq%252Easp%26Query%3DPage%253DRTQ%2526Symbol%253Dor
cl%2526REQUEST%255FMETHOD%253DGET&tw=43200&kv=2&ct=1014877682&ver=2.0.0248.1&
tpf=978415caf19b4eaa5a58f6ccd969d453 HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Host: login.passport.com
Pragma: no-cache
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Cookie: BrowserTest=Success?; MSPRequ=lt=1014877677&co=1&id=3

* Closing connection #0

Ended up at the same login page.

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
Received on 2002-02-28