Hello all,

Can anyone provide a different view on why cURL sends 2 extra cookies that don't belong (seem) to belong to the target domain and don't get sent when using a browser. I apologize for the long content to follow. It is my hope that someone will see the glitch. Btw, I'm an airline pilot, doing this for fun and to make computer life easier???

The background is that I use 3 php/curl pages to accomplish a task normally taking more steps and lots of useless data. The first page performs 13 curl transfers, to include login, and successfully gets the proper data. Selecting an item on this page requests the second page with only 1 curl transfer and also successfully gets the correct response. Then, requesting the 3rd page,which conducts 16 curl requests, there are 2 cookies that get sent, and it is my guess that this causes the server to respond differently. The primary emphasis is this help request is the presence of these 2 cookies being sent by curl. These two are:

portalNRTP=10%2F23%2F2008+8%3A57%3A29+AM; portalUser=00354611; (see the next block for where they are set)

They are received and set during 1st (ok) page as shown in the verbose output:

* Closing connection #2
* Connection (#2) was killed to make room (holds 6)
* About to connect() to www.jetnet.aa.com port 443 (#2)
* Trying 216.39.75.19... * connected
* Connected to www.jetnet.aa.com (216.39.75.19) port 443 (#2)
* successfully set certificate verify locations:
* CAfile: /usr/share/ssl/certs/ca-bundle.crt
  CApath: none
* SSL re-using session ID
* SSL connection using RC4-MD5
* Server certificate:
* subject: /C=US/ST=Texas/L=Fort Worth/O=American Airlines Inc./CN=www.jetnet.aa.com
* start date: 2008-05-13 13:27:16 GMT
* expire date: 2009-05-29 15:26:09 GMT
* common name: www.jetnet.aa.com (matched)
* issuer: /C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/CN=Thawte Premium Server CA/emailAddress=premium-server_at_thawte.com
* SSL certificate verify ok.
> POST /jetnet/admin/dologin.asp HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13
Host: www.jetnet.aa.com
Referer: https://www.jetnet.aa.com/jetnet/sso/sso.asp
Cookie: ASPSESSIONIDSARTBADQ=MINIFLBABMJKIPAAHKIHLOKL; SMSESSION=LBMOvPsLIcj4PqZZ...[omitted]...U8HwPW7RM; ASPSESSIONIDSCRTBBCQ=AGLOFLBAHMAFAOHFDPIKBKNA; SMIDENTITY=mKK82qk0n4j5k021.....[omitted].....ggtNzsylyu07j8xw==; FGNCLIID=febiucsmzq0pzpi5fsvzpxiswg4; BIGipServerPortal_80_Green=3624160778.20480.0000
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Content-Length: 22
Content-Type: application/x-www-form-urlencoded

< HTTP/1.1 302 Object moved
< Cache-control: private
< Content-Type: text/html
< Date: Thu, 23 Oct 2008 13:57:28 GMT
< Date: Thu, 23 Oct 2008 13:57:28 GMT
< Expires: Thu, 23 Oct 2008 13:56:28 GMT
< Location: ../communities/community.asp?CommunityID=203&intCurrentPageIndex=0&UserID=64953
< Server: Microsoft-IIS/5.0
< Server: Microsoft-IIS/5.0
* Replaced cookie SMSESSION="ukHumhb20......[omitted.....Fx3gLObI7" fo< Set-Cookie: SMSESSION=ukHumhb20.....Fx3gLObI7; path=/; domain=.aa.com
* Replaced cookie SMSESSION="ukHumhb20......[omitted.....Fx3gLObI7" fo< Set-Cookie: SMSESSION=ukHumhb20.....Fx3gLObI7; path=/; domain=.aa.com
* Added cookie GroupMemberships="GroupMem%3A%7B51%7D%7B201%7D%7B202%7D%7B363%7D%7B449%7D%7B598%7D%7B614%7D" for domain www.jetnet.aa.com, path /jetnet, expire 0
< Set-Cookie: GroupMemberships=GroupMem%3A%7B51%7D%7B201%7D%7B202%7D%7B363%7D%7B449%7D%7B598%7D%7B614%7D; path=/jetnet
 [omitted]
* Added cookie portalUser="00354611" for domain aa.com, path /, expire 0
< Set-Cookie: portalUser=00354611; domain=.aa.com; path=/
[omitted]
* Added cookie portalNRTP="10%2F23%2F2008+8%3A57%3A29+AM" for domain aa.com, path /, expire 0
< Set-Cookie: portalNRTP=10%2F23%2F2008+8%3A57%3A29+AM; domain=.aa.com; path=/
[omitted]
* Replaced cookie BIGipServerPortal_80_Green="3624160778.20480.0000" for domain www.jetnet.aa.com, path /, expire 1224777520
< Set-Cookie: BIGipServerPortal_80_Green=3624160778.20480.0000; expires=Thu, 23-Oct-2008 15:58:40 GMT; path=/
< X-Powered-By: ASP.NET
< X-Powered-By: ASP.NET
< Connection: Keep-Alive
< Content-Encoding: deflate
< Transfer-Encoding: chunked

Here is the curl setopt's for the beginning of the last and troubling page of the transfer (they change for 2-3 post requests) :

$header_array[] = "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
$header_array[] = "Accept-Language: en-us,en;q=0.5";
$header_array[] = "Accept-Encoding: gzip,deflate";
$header_array[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
$header_array[] = "Keep-Alive: 300";
$header_array[] = "Connection: keep-alive";

$ch1 = curl_init();

curl_setopt($ch1, CURLOPT_MAXCONNECTS, 16);
curl_setopt($ch1, CURLOPT_USERAGENT, $WEBBOT_NAME); // See above Go Stealthy
curl_setopt($ch1, CURLOPT_SSL_VERIFYPEER, FALSE); // Needed for https: & no certificate
curl_setopt($ch1, CURLOPT_TIMEOUT, CURL_TIMEOUT);
curl_setopt($ch1, CURLOPT_URL, $target); // Define target site
curl_setopt($ch1, CURLOPT_REFERER, $ref); // Define refering page
curl_setopt($ch1, CURLOPT_CONNECTTIMEOUT, 35);
   curl_setopt($ch1, CURLOPT_DEBUGFUNCTION, TRUE);
   curl_setopt($ch1, CURLOPT_VERBOSE, TRUE);
   curl_setopt($ch1, CURLOPT_STDERR, $fp);
curl_setopt($ch1, CURLOPT_NOPROGRESS, FALSE);
curl_setopt($ch1, CURLOPT_HTTPHEADER, $header_array); // Send Accept: Header values
curl_setopt($ch1, CURLOPT_RETURNTRANSFER, TRUE); // Return page in String
curl_setopt($ch1, CURLOPT_COOKIESESSION, FALSE);
curl_setopt($ch1, CURLOPT_COOKIEJAR, $cookie_file); // Where to WRITE cookies
curl_setopt($ch1, CURLOPT_COOKIEFILE, $cookie_file); // Where to READ cookies FROM
# curl_setopt($ch1, CURLOPT_COOKIE, $cookie); // Send specific cookie
curl_setopt($ch1, CURLOPT_HEADER, TRUE);
curl_setopt($ch1, CURLOPT_NOBODY, TRUE);
curl_setopt($ch1, CURLOPT_POST, FALSE); // Use GET Method

curl_setopt($ch1, CURLOPT_FRESH_CONNECT, TRUE); // Do NOT Force New Cache connection
# curl_setopt($ch1, CURLOPT_FOLLOWLOCATION, TRUE);
# curl_setopt($ch1, CURLOPT_ENCODING, "deflate"); // Attempt to Fix 'Content-Encoding: Deflate issue

And the verbose output from one example request:

* About to connect() to nrtp.jetnet.aa.com port 443 (#0)
* Trying 216.39.75.28... * connected
* Connected to nrtp.jetnet.aa.com (216.39.75.28) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: /usr/share/ssl/certs/ca-bundle.crt
  CApath: none
* SSL re-using session ID
* SSL connection using RC4-MD5
* Server certificate:
* subject: /C=US/ST=Texas/L=Fort Worth/O=American Airlines Inc./CN=nrtp.jetnet.aa.com
* start date: 2008-08-01 20:58:30 GMT
* expire date: 2010-08-01 20:58:30 GMT
* common name: nrtp.jetnet.aa.com (matched)
* issuer: /C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/CN=Thawte Premium Server CA/emailAddress=premium-server_at_thawte.com
* SSL certificate verify ok.
> POST /NonRev2App/booking_f.jsp HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13
Host: nrtp.jetnet.aa.com
Referer: https://nrtp.jetnet.aa.com/NonRev2App/list_front.jsp?bInternational=false&passengerNum=1
Cookie: JSESSIONID=0000LQfoEtxKh7NWnLbyWE4jvFr:134krsghb; portalNRTP=10%2F23%2F2008+8%3A57%3A29+AM; portalUser=00354611; SMSESSION=aokPyZ/LlXXu.....[ommitted].....mii0P0KHnFJ; SMIDENTITY=mKK82qk0n4j5k021.....[ommitted]...../UqggtNzsylyu07j8xw==; BIGipServerNRTP_YGP_80=880823818.20480.0000
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Content-Length: 190
Content-Type: application/x-www-form-urlencoded

100 190 0 0 0 190 0 176 --:--:-- 0:00:01 --:--:-- 176
100 190 0 0 0 190 0 91 --:--:-- 0:00:02 --:--:-- 0
100 190 0 0 0 190 0 61 --:--:-- 0:00:03 --:--:-- 0
100 190 0 0 0 190 0 46 --:--:-- 0:00:04 --:--:-- 0
100 190 0 0 0 190 0 37 --:--:-- 0:00:05 --:--:-- 0< HTTP/1.1 200 OK
< Date: Thu, 23 Oct 2008 13:59:02 GMT
< Server: IBM_HTTP_Server
< Content-Length: 697
< Keep-Alive: timeout=10, max=100
< Connection: Keep-Alive
< Content-Type: text/html;charset=ISO-8859-1
< Content-Language: en
* Replaced cookie BIGipServerNRTP_YGP_80="880823818.20480.0000" for domain nrtp.jetnet.aa.com, path /, expire 1224784747
< Set-Cookie: BIGipServerNRTP_YGP_80=880823818.20480.0000; expires=Thu, 23-Oct-2008 17:59:07 GMT; path=/
<

100 887 100 697 0 190 126 34 0:00:05 0:00:05 --:--:-- 114* Connection #0 to host nrtp.jetnet.aa.com left intact

The corresponding live HTTP headers output:

https://nrtp.jetnet.aa.com/NonRev2App/booking_f.jsp

POST /NonRev2App/booking_f.jsp HTTP/1.1
Host: nrtp.jetnet.aa.com
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16 FirePHP/0.1.2
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: https://nrtp.jetnet.aa.com/NonRev2App/list_front.jsp?bInternational=false&passengerNum=1
Cookie: SMIDENTITY=GATyXfQkfjQb0ifk.....[ommitted].....J2pyCpJbsI41H4b8vi; BIGipServerNRTP_YGP_80=880823818.20480.0000; SMSESSION=xthlBYJmRvVxaUgpGu.....[ommitted].....Zaswzjg9mRvdPcnlO/L+o; JSESSIONID=0000oGKiY2hKIs8nO5s5Fd4nZ0t:134krsaai
Content-Type: application/x-www-form-urlencoded
Content-Length: 190
passengerNum=1&bInternational=false&seatType_1=Y&tripDescription=JAX+%2F+MIA+-+09%2F27%2F2008&passenger_1=1&priority_1=D2&ssr_1=x&phone1=305+394-xxxx&phone1Type=HOME&phone2=0&phone2Type=HOME

HTTP/1.x 200 OK
Date: Thu, 25 Sep 2008 19:01:58 GMT
Server: IBM_HTTP_Server
Set-Cookie: SMSESSION=RFc7KXBa/Osqm3b.....[ommitted].....wOwJSc3lu8xg+H4/DxcaAE0; path=/; domain=.aa.com
Set-Cookie: BIGipServerNRTP_YGP_80=880823818.20480.0000; expires=Thu, 25-Sep-2008 23:02:00 GMT; path=/
Content-Length: 673
Keep-Alive: timeout=10, max=100
Connection: Keep-Alive
Content-Type: text/html;charset=ISO-8859-1
Content-Language: en
-

Thanks for looking. I've been looking at this for sooo long. No suggestions will be overlooked.

David Colter

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Received on 2008-10-24