Perhaps the proxy is pre-fetching and caching the embedded content to
make your Internet experience "faster"?...

Not sure why it would do so only when using cURL and not when using FF...

Could you take the proxy out of the picture temporarily to test?

On Thu, January 17, 2008 1:26 pm, hugo wrote:
> i'm a newbie, and i'm having a difficult time trying to figure out
> the behaviour of my php curl script when accessing a particular site.
>
> here is my program:
>
> <?
> $cookiefile = "cookie.txt";
> $proxy = "proxy.shr.secureserver.net:3128";
> $url = "https://www.usfsaonline.org";
>
> $ch = curl_init();
> curl_setopt($ch, CURLOPT_URL, $url );
> curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
> curl_setopt($ch, CURLOPT_PROXY,$proxy);
> curl_setopt ($ch, CURLOPT_HTTPPROXYTUNNEL, true);
> curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile );
> curl_setopt($ch, CURLOPT_HEADER, 1);
> curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );
> curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false );
> $result =curl_exec ($ch);
> echo $result;
> ?>
>
> now when i log into the usfsaonline.org site with my firefox browser
> Live HTTP Headers gives me the following:
>
>> http://www.usfsaonline.org/
>>
>> GET / HTTP/1.1
>> Host: www.usfsaonline.org
>> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:
>> 1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
>> Accept: text/xml,application/xml,application/xhtml+xml,text/
>> html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
>> Accept-Language: en-us,en;q=0.5
>> Accept-Encoding: gzip,deflate
>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>> Keep-Alive: 300
>> Connection: keep-alive
>> Cookie: ASPSESSIONIDAABBSTQD=IMBFJMJDEPDCFGBGEOJGOAPF
>>
>> HTTP/1.x 302 Object moved
>> Date: Thu, 17 Jan 2008 19:10:30 GMT
>> Server: Microsoft-IIS/6.0
>> X-Powered-By: ASP.NET
>> Location: https://www.usfsaonline.org/IntraLock/Login.asp?
>> targetpath=%2FDefault%2Easp
>> Content-Length: 196
>> Content-Type: text/html
>> Cache-Control: private
>> ----------------------------------------------------------
>>
>> https://www.usfsaonline.org/IntraLock/Login.asp?targetpath=%
>> 2FDefault%2Easp
>>
>> GET /IntraLock/Login.asp?targetpath=%2FDefault%2Easp HTTP/1.1
>> Host: www.usfsaonline.org
>> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:
>> 1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
>> Accept: text/xml,application/xml,application/xhtml+xml,text/
>> html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
>> Accept-Language: en-us,en;q=0.5
>> Accept-Encoding: gzip,deflate
>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>> Keep-Alive: 300
>> Connection: keep-alive
>> Cookie: ASPSESSIONIDAABBSTQD=IMBFJMJDEPDCFGBGEOJGOAPF
>>
>>
>> HTTP/1.x 200 OK
>> Date: Thu, 17 Jan 2008 19:10:31 GMT
>> Server: Microsoft-IIS/6.0
>> X-Powered-By: ASP.NET
>> Content-Length: 53442
>> Content-Type: text/html
>> Cache-Control: private
>>
>> ----------------------------------------------------------
>>
>
> but when i execute my php script through the web page it is on
> ( http://atlantaopen.info/usfsa.php ) Live HTTP Headers comes back
> with a lot of 404 errors that reflect GETs that i never asked for or
> issued and that are actually requests for information that is in the
> page source of the www.usfsaonline.org site, and not in my code at
> all.
> This is driving me crazy... does anyone know what i'm doing wrong?
>
> ( here is truncated output from Live HTTP Headers for my php curl
> script )
>
>
>> http://atlantaopen.info/usfsa.php
>>
>> GET /usfsa.php HTTP/1.1
>> Host: atlantaopen.info
>> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:
>> 1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
>> Accept: text/xml,application/xml,application/xhtml+xml,text/
>> html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
>> Accept-Language: en-us,en;q=0.5
>> Accept-Encoding: gzip,deflate
>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>> Keep-Alive: 300
>> Connection: keep-alive
>> Cookie: 2c9086f9455d6335b54958f99919592a=j2k8cgeaj4c0cihnru7pu0oj54
>>
>> HTTP/1.x 200 OK
>> Date: Thu, 17 Jan 2008 19:17:04 GMT
>> Server: Apache
>> Keep-Alive: timeout=15, max=100
>> Connection: Keep-Alive
>> Transfer-Encoding: chunked
>> Content-Type: text/html
>> ----------------------------------------------------------
>> http://atlantaopen.info/media/navs/OnlineEventReg2.gif
>>
>> GET /media/navs/OnlineEventReg2.gif HTTP/1.1
>> Host: atlantaopen.info
>> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:
>> 1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
>> Accept: image/png,*/*;q=0.5
>> Accept-Language: en-us,en;q=0.5
>> Accept-Encoding: gzip,deflate
>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>> Keep-Alive: 300
>> Connection: keep-alive
>> Referer: http://atlantaopen.info/usfsa.php
>> Cookie: 2c9086f9455d6335b54958f99919592a=j2k8cgeaj4c0cihnru7pu0oj54
>>
>>
>> HTTP/1.x 404 Not Found
>> Date: Thu, 17 Jan 2008 19:17:08 GMT
>> Server: Apache
>> Keep-Alive: timeout=15, max=99
>> Connection: Keep-Alive
>> Transfer-Encoding: chunked
>> Content-Type: text/html
>> ----------------------------------------------------------
>>
>> http://atlantaopen.info/media/navs/OnlineEventReg1.gif
>>
>> GET /media/navs/OnlineEventReg1.gif HTTP/1.1
>> Host: atlantaopen.info
>> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:
>> 1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
>> Accept: image/png,*/*;q=0.5
>> Accept-Language: en-us,en;q=0.5
>> Accept-Encoding: gzip,deflate
>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>> Keep-Alive: 300
>> Connection: keep-alive
>> Referer: http://atlantaopen.info/usfsa.php
>> Cookie: 2c9086f9455d6335b54958f99919592a=j2k8cgeaj4c0cihnru7pu0oj54
>>
>> HTTP/1.x 404 Not Found
>> Date: Thu, 17 Jan 2008 19:17:09 GMT
>> Server: Apache
>> Keep-Alive: timeout=15, max=98
>> Connection: Keep-Alive
>> Transfer-Encoding: chunked
>> Content-Type: text/html
>> ----------------------------------------------------------
>> http://atlantaopen.info/media/spacer.gif
>>
>> GET /media/spacer.gif HTTP/1.1
>> Host: atlantaopen.info
>> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:
>> 1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
>> Accept: image/png,*/*;q=0.5
>> Accept-Language: en-us,en;q=0.5
>> Accept-Encoding: gzip,deflate
>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>> Keep-Alive: 300
>> Connection: keep-alive
>> Referer: http://atlantaopen.info/usfsa.php
>> Cookie: 2c9086f9455d6335b54958f99919592a=j2k8cgeaj4c0cihnru7pu0oj54
>>
>> HTTP/1.x 404 Not Found
>> Date: Thu, 17 Jan 2008 19:17:09 GMT
>> Server: Apache
>> Keep-Alive: timeout=15, max=98
>> Connection: Keep-Alive
>> Transfer-Encoding: chunked
>> Content-Type: text/html
>> ----------------------------------------------------------
>> http://atlantaopen.info/media/shield_top.gif
>>
>> GET /media/shield_top.gif HTTP/1.1
>> Host: atlantaopen.info
>> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:
>> 1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
>> Accept: image/png,*/*;q=0.5
>> Accept-Language: en-us,en;q=0.5
>> Accept-Encoding: gzip,deflate
>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>> Keep-Alive: 300
>> Connection: keep-alive
>> Referer: http://atlantaopen.info/usfsa.php
>> Cookie: 2c9086f9455d6335b54958f99919592a=j2k8cgeaj4c0cihnru7pu0oj54
>>
>> HTTP/1.x 404 Not Found
>> Date: Thu, 17 Jan 2008 19:17:09 GMT
>> Server: Apache
>> Keep-Alive: timeout=15, max=97
>> Connection: Keep-Alive
>> Transfer-Encoding: chunked
>> Content-Type: text/html
>> ----------------------------------------------------------
>> http://atlantaopen.info/media/USFSA.gif
>>
>> GET /media/USFSA.gif HTTP/1.1
>> Host: atlantaopen.info
>> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:
>> 1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
>> Accept: image/png,*/*;q=0.5
>> Accept-Language: en-us,en;q=0.5
>> Accept-Encoding: gzip,deflate
>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>> Keep-Alive: 300
>> Connection: keep-alive
>> Referer: http://atlantaopen.info/usfsa.php
>> Cookie: 2c9086f9455d6335b54958f99919592a=j2k8cgeaj4c0cihnru7pu0oj54
>>
>> HTTP/1.x 404 Not Found
>> Date: Thu, 17 Jan 2008 19:17:09 GMT
>> Server: Apache
>> Keep-Alive: timeout=15, max=97
>> Connection: Keep-Alive
>> Transfer-Encoding: chunked
>> Content-Type: text/html
>>
>> ----------------------------------------------------------
>> http://atlantaopen.info/media/newLogoUSFSA.gif
>>
>> GET /media/newLogoUSFSA.gif HTTP/1.1
>> Host: atlantaopen.info
>> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:
>> 1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
>> Accept: image/png,*/*;q=0.5
>> Accept-Language: en-us,en;q=0.5
>> Accept-Encoding: gzip,deflate
>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>> Keep-Alive: 300
>> Connection: keep-alive
>> Referer: http://atlantaopen.info/usfsa.php
>> Cookie: 2c9086f9455d6335b54958f99919592a=j2k8cgeaj4c0cihnru7pu0oj54
>>
>> HTTP/1.x 404 Not Found
>> Date: Thu, 17 Jan 2008 19:17:09 GMT
>> Server: Apache
>> Keep-Alive: timeout=15, max=96
>> Connection: Keep-Alive
>> Transfer-Encoding: chunked
>> Content-Type: text/html
>> ----------------------------------------------------------
>> http://atlantaopen.info/media/navs/Competitions2.gif
>>
>> GET /media/navs/Competitions2.gif HTTP/1.1
>> Host: atlantaopen.info
>> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:
>> 1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
>> Accept: image/png,*/*;q=0.5
>> Accept-Language: en-us,en;q=0.5
>> Accept-Encoding: gzip,deflate
>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>> Keep-Alive: 300
>> Connection: keep-alive
>> Referer: http://atlantaopen.info/usfsa.php
>> Cookie: 2c9086f9455d6335b54958f99919592a=j2k8cgeaj4c0cihnru7pu0oj54
>>
>> HTTP/1.x 404 Not Found
>> Date: Thu, 17 Jan 2008 19:17:09 GMT
>> Server: Apache
>> Keep-Alive: timeout=15, max=96
>> Connection: Keep-Alive
>> Transfer-Encoding: chunked
>> Content-Type: text/html
>> ----------------------------------------------------------
>>
>
>
> ........... etc. etc. etc ...........
>
>
>> ----------------------------------------------------------
>> http://atlantaopen.info/media/navs/viewUpload.gif
>>
>> GET /media/navs/viewUpload.gif HTTP/1.1
>> Host: atlantaopen.info
>> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:
>> 1.8.1.11) Gecko/20071127 Firefox/2.0.0.11
>> Accept: image/png,*/*;q=0.5
>> Accept-Language: en-us,en;q=0.5
>> Accept-Encoding: gzip,deflate
>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>> Keep-Alive: 300
>> Connection: keep-alive
>> Referer: http://atlantaopen.info/usfsa.php
>> Cookie: 2c9086f9455d6335b54958f99919592a=j2k8cgeaj4c0cihnru7pu0oj54
>>
>> HTTP/1.x 404 Not Found
>> Date: Thu, 17 Jan 2008 19:17:14 GMT
>> Server: Apache
>> Keep-Alive: timeout=15, max=95
>> Connection: Keep-Alive
>> Transfer-Encoding: chunked
>> Content-Type: text/html
>> ----------------------------------------------------------
>>
>>
> _______________________________________________
> http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
>

-- 
Some people have a "gift" link here.
Know what I want?
I want you to buy a CD from some indie artist.
http://cdbaby.com/from/lynch
Yeah, I get a buck. So?
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php