cURL / Mailing Lists / curl-users / Single Mail

curl-users

Re: cURL & Monster.com

From: B Z <bz029_at_hotmail.com>
Date: Thu, 14 Jul 2005 10:09:21 -0700

>Rather than guessing, why not use Live Headers to see the actual request that
>Firefox sends for the query. That will show you exactly what you will need
>to emulate using a curl command. Once you've accomplished that, you can then
>start figuring out whether you'll need to parse Javascript to get that
>data.

Thanks for the advice. I was unaware of that very useful tool.

So - what your saying is i should have my curl command visit all the specified URLS that show up in the Live Headers log?

Thanks again for the help!

Here is my log:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

http://www.monster.com/

GET / HTTP/1.1
Host: www.monster.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: ez=95101-95103%2b95106%2b95108-95139%2b95141-95142%2b95148%2b95150-95161%2b95164%2b95170-95173%2b95190-95194%2b95196; mnshp3=2; COOKIE_ID=67.172.177.197-426655488.29722772
If-Modified-Since: Thu, 14 Jul 2005 16:53:17 GMT

HTTP/1.x 200 OK
Server: Microsoft-IIS/6.0
Last-Modified: Thu, 14 Jul 2005 16:53:25 GMT
Content-Length: 46120
Content-Type: text/html
Set-Cookie: ez=95101-95103%2b95106%2b95108-95139%2b95141-95142%2b95148%2b95150-95161%2b95164%2b95170-95173%2b95190-95194%2b95196; expires=Tue, 10 Jan 2006 16:53:25 GMT; path=/; domain=.monster.com
Set-Cookie: mnshp3=2; expires=Tue, 13 Dec 2005 18:53:25 GMT; path=/; domain=www.monster.com;
Expires: Thu, 14 Jul 2005 16:53:25 GMT
Cache-Control: max-age=0, no-cache
Pragma: no-cache
Date: Thu, 14 Jul 2005 16:53:25 GMT
Connection: keep-alive
----------------------------------------------------------
http://media.monster.com/mm/usen/home/pixel.gif

GET /mm/usen/home/pixel.gif HTTP/1.1
Host: media.monster.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5
Accept: image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.monster.com/
Cookie: ez=95101-95103%2b95106%2b95108-95139%2b95141-95142%2b95148%2b95150-95161%2b95164%2b95170-95173%2b95190-95194%2b95196; COOKIE_ID=67.172.177.197-426655488.29722772

HTTP/1.x 200 OK
Server: Microsoft-IIS/5.0
P3P: policyref="http://media.monster.com/w3c/p3p.xml", CP="NOI DSP COR NID ADM DEV PSA OUR IND UNI PUR COM NAV INT STA"
Content-Length: 463
Content-Type: text/html
Cache-Control: private, max-age=0
Date: Thu, 14 Jul 2005 16:53:25 GMT
Connection: keep-alive
----------------------------------------------------------
http://ads.monster.com/html.ng/site=mons&affiliate=mons&app=www&size=168x68&pp=1

GET /html.ng/site=mons&affiliate=mons&app=www&size=168x68&pp=1 HTTP/1.1
Host: ads.monster.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.monster.com/
Cookie: ez=95101-95103%2b95106%2b95108-95139%2b95141-95142%2b95148%2b95150-95161%2b95164%2b95170-95173%2b95190-95194%2b95196; COOKIE_ID=67.172.177.197-426655488.29722772; NGUserID=a320aa6-1944-1121359811-9

GET /html.ng/site=mons&affiliate=mons&app=www&size=144x63&pp=5 HTTP/1.1
Host: ads.monster.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.monster.com/
Cookie: ez=95101-95103%2b95106%2b95108-95139%2b95141-95142%2b95148%2b95150-95161%2b95164%2b95170-95173%2b95190-95194%2b95196; COOKIE_ID=67.172.177.197-426655488.29722772; NGUserID=a320aa6-1944-1121359811-9

HTTP/1.x 200 OK
Server: Microsoft-IIS/5.0
Date: Thu, 14 Jul 2005 16:53:25 GMT
P3P: policyref="http://media.monster.com/w3c/p3p.xml", CP="NOI DSP COR NID ADM DEV PSA OUR IND UNI PUR COM NAV INT STA"
Connection: close
Content-Type: text/html
----------------------------------------------------------
http://cookie.monster.com/DCS000003_6D4Q/dcs.gif?&dcsdat=1121417991078&dcssip=www.monster.com&dcsuri=/&WT.tz=-7&WT.bh=1&WT.ul=en-US&WT.cd=32&WT.sr=1024x768&WT.jo=Yes&WT.ti=Monster%20Jobs%20-%20Get%20work.%20Network.%20Build%20a%20better%20career.%20Today's%20the%20day.&WT.js=Yes&WT.jv=1.5&WT.cg_n=home&WT.mns_tc_wwhp=trump2

GET /DCS000003_6D4Q/dcs.gif?&dcsdat=1121417991078&dcssip=www.monster.com&dcsuri=/&WT.tz=-7&WT.bh=1&WT.ul=en-US&WT.cd=32&WT.sr=1024x768&WT.jo=Yes&WT.ti=Monster%20Jobs%20-%20Get%20work.%20Network.%20Build%20a%20better%20career.%20Today's%20the%20day.&WT.js=Yes&WT.jv=1.5&WT.cg_n=home&WT.mns_tc_wwhp=trump2 HTTP/1.1
Host: cookie.monster.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5
Accept: image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.monster.com/
Cookie: ez=95101-95103%2b95106%2b95108-95139%2b95141-95142%2b95148%2b95150-95161%2b95164%2b95170-95173%2b95190-95194%2b95196; COOKIE_ID=67.172.177.197-426655488.29722772

HTTP/1.x 200 OK
Content-Length: 43
Content-Type: image/gif
Last-Modified: Mon, 28 Jan 2002 14:51:42 GMT
Accept-Ranges: bytes
Etag: "013394cba8c11:4a7"
Server: Microsoft-IIS/6.0
Set-Cookie: COOKIE_ID=zzz;expires=Thu, 1 Jan 1970 00:00:00 GMT;path=/;domain=cookie.monster.com;
P3P: CP="NOI DSP COR NID ADM DEV PSA OUR IND UNI PUR COM NAV INT STA",policyref="http://media.monster.com/w3c/p3p.xml"
Date: Thu, 14 Jul 2005 16:53:26 GMT
Connection: close
----------------------------------------------------------
http://ad.doubleclick.net/activity;src=870983;type=consu801;cat=homep810;ord=3676422349986.125?

GET /activity;src=870983;type=consu801;cat=homep810;ord=3676422349986.125? HTTP/1.1
Host: ad.doubleclick.net
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5
Accept: image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.monster.com/

HTTP/1.x 302 Moved Temporarily
Content-Length: 0
Location: http://ad.doubleclick.net/activity;src=870983;type=consu801;cat=homep810;ord=3676422349986.125?&_dc_ck=try
Set-Cookie: test_cookie=CheckForPermission; path=/; domain=.doubleclick.net; expires=Thu, 14 Jul 2005 17:08:26 GMT
P3P: CP="CURa ADMa DEVa PSAo PSDo OUR BUS UNI PUR INT DEM STA PRE COM NAV OTC NOI DSP COR"
----------------------------------------------------------
http://switch.atdmt.com/nyc/action/Monster_Homepage

GET /nyc/action/Monster_Homepage HTTP/1.1
Host: switch.atdmt.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5
Accept: image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.monster.com/
Cookie: AA002=1121359812-704645812/1122569443
If-Modified-Since: Fri, 23 Mar 2001 19:01:15 GMT
If-None-Match: "90aee1a2cbb3c01:28c"

HTTP/1.x 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 42
Content-Type: image/gif
Content-Location: http://image.atdmt.com/images/pixel.gif
Expires: 0
Last-Modified: Fri, 23 Mar 2001 19:01:15 GMT
Accept-Ranges: bytes
Etag: "90aee1a2cbb3c01:288"
Server: Microsoft-IIS/6.0
D-Type: M
Connection: close, close
Date: Thu, 14 Jul 2005 16:53:26 GMT
----------------------------------------------------------
http://switch.atdmt.com/action/deu_monster_home1

GET /action/deu_monster_home1 HTTP/1.1
Host: switch.atdmt.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5
Accept: image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.monster.com/
Cookie: AA002=1121359812-704645812/1122569443
If-Modified-Since: Fri, 23 Mar 2001 19:01:15 GMT
If-None-Match: "90aee1a2cbb3c01:28c"

HTTP/1.x 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 42
Content-Type: image/gif
Content-Location: http://image.atdmt.com/images/pixel.gif
Expires: 0
Last-Modified: Fri, 23 Mar 2001 19:01:15 GMT
Accept-Ranges: bytes
Etag: "90aee1a2cbb3c01:282"
Server: Microsoft-IIS/6.0
D-Type: M
Connection: close, close
Date: Thu, 14 Jul 2005 16:53:26 GMT
----------------------------------------------------------
http://ads.monster.com/html.ng/site=mons&affiliate=mons&app=www&size=144x63&pp=1

GET /html.ng/site=mons&affiliate=mons&app=www&size=144x63&pp=1 HTTP/1.1
Host: ads.monster.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.monster.com/
Cookie: ez=95101-95103%2b95106%2b95108-95139%2b95141-95142%2b95148%2b95150-95161%2b95164%2b95170-95173%2b95190-95194%2b95196; COOKIE_ID=67.172.177.197-426655488.29722772; NGUserID=a320aa6-1944-1121359811-9

GET /html.ng/site=mons&affiliate=mons&app=www&size=144x63&pp=2 HTTP/1.1
Host: ads.monster.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.monster.com/
Cookie: ez=95101-95103%2b95106%2b95108-95139%2b95141-95142%2b95148%2b95150-95161%2b95164%2b95170-95173%2b95190-95194%2b95196; COOKIE_ID=67.172.177.197-426655488.29722772; NGUserID=a320aa6-1944-1121359811-9

GET /activity;src=870983;type=consu801;cat=homep810;ord=3676422349986.125?&_dc_ck=try HTTP/1.1
Host: ad.doubleclick.net
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.monster.com/

HTTP/1.x 200 OK
Content-Type: image/gif
Pragma: no-cache
Cache-Control: private, no-cache
Content-Length: 43
----------------------------------------------------------
http://media.monster.com/ads/usen/house/featuredemp/b2b_inside_sales2.gif

GET /ads/usen/house/featuredemp/b2b_inside_sales2.gif HTTP/1.1
Host: media.monster.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.9) Gecko/20050711 Firefox/1.0.5
Accept: image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://ads.monster.com/html.ng/site=mons&affiliate=mons&app=www&size=144x63&pp=1
Cookie: ez=95101-95103%2b95106%2b95108-95139%2b95141-95142%2b95148%2b95150-95161%2b95164%2b95170-95173%2b95190-95194%2b95196; COOKIE_ID=67.172.177.197-426655488.29722772
If-Modified-Since: Wed, 13 Jul 2005 19:49:49 GMT
If-None-Match: "512d87e487c51:88f"

HTTP/1.x 304 Not Modified
Content-Type: image/gif
Last-Modified: Wed, 13 Jul 2005 19:49:49 GMT
Etag: "512d87e487c51:88f"
Date: Thu, 14 Jul 2005 16:53:26 GMT
Connection: keep-alive
----------------------------------------------------------

Received on 2005-07-14

This message: [ Message body ]
Next message: Dan Fandrich: "Re: cURL & Monster.com"
Previous message: Dan Fandrich: "Re: cURL & Monster.com"
In reply to: Dan Fandrich: "Re: cURL & Monster.com"
Next in thread: Dan Fandrich: "Re: cURL & Monster.com"
Reply: Dan Fandrich: "Re: cURL & Monster.com"

Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]