Hi,

I am trying to retrieve a file from a website using
Php/curl. I believe I am setting the required session
ID cookies correctly by visiting a page which sets an
ASPSESSIONID cookie, but next curl only retrieves the
first 2,490 bytes of the 21.06 kB target file. It
tests OK in firefox, and if I delete the cookies in
Firefox I see the same problem (only the first 2,490
bytes are received) which makes me believe it is
cookie related. The only other thing I thought it
could be is the chunked encoding that is used. The
website is running Apache. I have never seen this
behaivour before, is it common? Can anyone help from
the info below?

My system configuration:
PHP Version 4.3.11
Curl information: libcurl/7.12.0 OpenSSL/0.9.7a
zlib/1.1.4

Other observations:
The 2 files I retrieve (1 to set cookie, 1 target
file) appear to come from different server types
(Apache and Microsoft-IIS/5.0). See headers below.

My code:

$fh = fopen('curl.out','a') or die($php_errormsg);
$ch = curl_init();
// Connect to page to set session cookie
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt");
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_REFERER,
"http://www.site.com");
curl_setopt($ch, CURLOPT_URL,
"http://www.site.com/afm.asp");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_HEADER, 1);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_STDERR, $fh); // where to log
$curl_response=curl_exec($ch);
curl_close($ch);
unset($ch);
fclose($fh) or die($php_errormsg);

//Now retrieve target page after setting session
cookie
$fh = fopen('curl.out','a') or die($php_errormsg);
$ch = curl_init();
curl_setopt($ch, CURLOPT_COOKIEFILE,
"flybecookies.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR,
"flybecookies.txt");
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_REFERER,
"http://www.site.com");
curl_setopt($ch, CURLOPT_ENCODING, "gzip,deflate");
curl_setopt($ch, CURLOPT_URL,
"http://www.site.com/page2.do");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_HEADER, 1);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_STDERR, $fh); // where to log
$curl_response=curl_exec($ch);
curl_close($ch);
unset($ch);
fclose($fh) or die($php_errormsg);

Firefox headers:
http://www.site.com/afm.asp

GET /afm.asp HTTP/1.1
Host: www.site.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-GB; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1
Accept: */*
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.site.com/

HTTP/1.x 200 OK
Set-Cookie:
ASPSESSIONIDCSARSQCR=MHHDMMBBMGGMMDIOJHDAPKAO; path=/
Cache-Control: max-age=0, private
Content-Length: 568
Date: Fri, 24 Feb 2006 20:45:55 GMT
Server: Microsoft-IIS/5.0
Content-Type: application/x-javascript
Backend-Node: 128.201.1.20:80

http://www.site.com/Javascript.do

GET /Javascript.do HTTP/1.1
Host: www.site.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-GB; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1
Accept: */*
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.site.com/
Cookie: ASPSESSIONIDCSARSQCR=MHHDMMBBMGGMMDIOJHDAPKAO

HTTP/1.x 200 OK
Content-Encoding: gzip
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie:
JSESSIONID=5BCE0E61A6F7C5FB39369D0C56BD27E1; Path=/cam
Set-Cookie:
K-JSESSIONID-0x6b55515b=E05ACD40467B36B3184A49107399A7D0
Cache-Control: max-age=0, private
Date: Fri, 24 Feb 2006 20:42:52 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Server: Apache
Content-Type: text/javascript;charset=UTF-8
Backend-Node: whiskey:80
Keep-Alive: timeout=15, max=67
Pragma: No-cache

And from my Curl verbose output for the same page
requests:

* About to connect() to www.site.com port 80
* Connected to www.site.com (123.456.789.123) port 80
> GET /afm.asp HTTP/1.1
Host: www.site.com
Pragma: no-cache
Accept: */*
Referer: http://www.site.com

< HTTP/1.1 200 OK
* Added cookie
ASPSESSIONIDCSARSQCR="NHGANMBBOFPEMDMMPOOCPPCL" for
domain www.site.com, path /, expire 0
< Set-Cookie:
ASPSESSIONIDCSARSQCR=NHGANMBBOFPEMDMMPOOCPPCL; path=/
< Cache-Control: max-age=0, private
< Content-Length: 568
< Date: Sun, 26 Feb 2006 11:32:32 GMT
< Server: Microsoft-IIS/5.0
< Content-Type: application/x-javascript
< Backend-Node: 128.201.1.20:80
* Connection #0 to host www.site.com left intact
* Closing connection #0

* About to connect() to www.site.com port 80
* Connected to www.site.com (123.456.789.123) port 80
> GET /Javascript.do HTTP/1.1
Host: www.site.com
Pragma: no-cache
Accept: */*
Referer: http://www.site.com
Cookie: ASPSESSIONIDCSARSQCR=NHGANMBBOFPEMDMMPOOCPPCL

< HTTP/1.1 200 OK
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
* Added cookie
JSESSIONID="7935EF2CE42D8799ADD49AAABA4F7396" for
domain www.flybe.com, path /cam, expire 0
< Set-Cookie:
JSESSIONID=7935EF2CE42D8799ADD49AAABA4F7396; Path=/cam
* Added cookie
K-JSESSIONID-0x6b55515b="ADB084A9541610ECE7C81D6325300A9D"
for domain www.flybe.com, path /cam/, expire 0
< Set-Cookie:
K-JSESSIONID-0x6b55515b=ADB084A9541610ECE7C81D6325300A9D
< Cache-Control: max-age=0, private
< Date: Sun, 26 Feb 2006 11:29:28 GMT
< Transfer-Encoding: chunked
< Server: Apache
< Content-Type: text/javascript;charset=UTF-8
< Backend-Node: tango:80
< Pragma: No-cache
* Connection #0 to host www.site.com left intact
* Closing connection #0

Regards,
Martin.

                
___________________________________________________________
To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Received on 2006-02-27