Hi Waruna,

thanks for your answer I really appreciate your help.
Sorry for the late answer but the form from which I’m trying to retrieve
data was not working during the last 2 days, so not way to test any efforts.
I tried giving as a parameter the array but its not working I'm getting a
header "400 Bad request" and html message "Your browser sent a query this
server could not understand".
Using a string for the form parameters Im able to get a correct header
response from server but still same problem with the form submission, so
I’m not able to retrieve any data records from server.

Below I exposed all the headers using the browser to submit successfully
the form so you can see how how headers are formed when form submission is
successful. Also I added all the PHP-cURL code that I’m using to try to
retrieve the data from the form and which header I’m getting during the
execution. About the PHP-cURL execution process I exposed two variant of
the code, one using array and other using string for the Form data.

I will really appreciate any clue

*## Process using Chromium Web browser
*

*(1) LOAD FORM*

*Request Headers*
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cache-Control: max-age=0
Connection: keep-alive
Cookie:
JSESSIONID=FbjYSL9dMpjk6hv8cMYLryW1bZ6nbLJ4qVLmKLxRjPLMbSplYrpK!806951883;
BIGipServerpl-prod_iplanet_http_v4=214476992.20480.0000;
__utma=217352402.2008711257.1376271515.1376271515.1376271515.1;
__utmb=217352402.2.10.1376271515; __utmc=217352402;
__utmz=217352402.1376271515.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Host: www.ura.gov.sg
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like
Gecko) Ubuntu Chromium/28.0.1500.71 Chrome/28.0.1500.71 Safari/537.36
*
Response Headers*
Connection:Keep-Alive
Content-Encoding:gzip
Content-Length:31731
Content-type:text/html;charset=UTF-8
Date:Mon, 12 Aug 2013 02:02:00 GMT
Vary:Accept-Encoding
X-Powered-By:Servlet/2.4 JSP/2.0
X-wily-info:Clear guid=0
X-wily-servlet:Encrypt1
WSry9LDfbZEeaCzdlYhs5ZVGfvNRF+pLLMu9RzcTB5ahtwuSbo0msWybnDTnQk5ZkYb9ckzGENTW6lIEDCgt6lqePNgpd97zx8P6moDyz/JNd2YpuT2S7AH3HKPkIG0OwcDaeqJQOtl7kaHL9+tPLqAAvMyPvJW75zRzt4sHDP4aX0LYJEdhdttHVd+0Y6pH

*(2) FORM SUBMISSION*

*Request Headers*
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:303
Content-Type:application/x-www-form-urlencoded
Cookie:JSESSIONID=FbjYSL9dMpjk6hv8cMYLryW1bZ6nbLJ4qVLmKLxRjPLMbSplYrpK!806951883;
BIGipServerpl-prod_iplanet_http_v4=214476992.20480.0000;
__utma=217352402.2008711257.1376271515.1376271515.1376271515.1;
__utmb=217352402.3.10.1376271515; __utmc=217352402;
__utmz=217352402.1376271515.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Host:www.ura.gov.sg
Origin:https://www.ura.gov.sg
Referer:https://www.ura.gov.sg/realEstateIIWeb/resiRental/search.action
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like
Gecko) Ubuntu Chromium/28.0.1500.71 Chrome/28.0.1500.71 Safari/537.36

*Form Data*
submissionType=pd&from_Date_Prj=JUN-2013&to_Date_Prj=JUN-2013&__multiselect_projectNameList=&__multiselect_selectedProjects=&propertyType=lp&from_Date=JUN-2013&to_Date=JUN-2013&__multiselect_postalDistrictList=&selectedPostalDistricts=01&selectedPostalDistricts=02&__multiselect_selectedPostalDistricts=

*Response Headers*
Connection:Keep-Alive
Content-Encoding:gzip
Content-Length:3615
Content-type:text/html;charset=UTF-8
Date:Mon, 12 Aug 2013 02:05:07 GMT
Vary:Accept-Encoding
X-Powered-By:Servlet/2.4 JSP/2.0
X-wily-info:Clear guid=0
X-wily-servlet:Encrypt1
WSry9LDfbZEeaCzdlYhs5ZVGfvNRF+pLLMu9RzcTB5ahtwuSbo0msWybnDTnQk5ZkYb9ckzGENTW6lIEDCgt6lqePNgpd97zx8P6moDyz/I9nEzIbgUwB5XGtDorgEVzWOi6vfH2yTz2QxTnJgl6ygXO1eu2FOcWzQjmEx4awzaoZjgDn2fouL1iN9jGfYYM

*## Process using cURL PHP *(with array for parameters)*
*

*Code to load the Form*

$ch = curl_init();

$GET_headers = array(
    'Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language: en-US,en;q=0.8',
    'Connection: keep-alive',
    'Host: www.ura.gov.sg',
    'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML,
like Gecko) Ubuntu Chromium/28.0.1500.71 Chrome/28.0.1500.71 Safari/537.36'
    );

// first request to collect the cookies
curl_setopt($ch, CURLOPT_ENCODING, "gzip,deflate,sdch");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 1);
curl_setopt($ch, CURLOPT_URL, "
https://www.ura.gov.sg/realEstateIIWeb/resiRental/search.action");
curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, $GET_headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
$result = curl_exec($ch);
preg_match_all('/^Set-Cookie:\s*([^;]*)/mi', $result, $m);

*Request Headers
*
GET /realEstateIIWeb/resiRental/search.action HTTP/1.1
Accept-Encoding: gzip,deflate,sdch
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.8
Connection: keep-alive
Host: www.ura.gov.sg
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like
Gecko) Ubuntu Chromium/28.0.1500.71 Chrome/28.0.1500.71 Safari/537.36

*Response Headers
*

< HTTP/1.1 200 OK
< Date: Sat, 10 Aug 2013 11:05:55 GMT
< Content-type: text/html;charset=UTF-8
< Cache-Control: no-cache="set-cookie"
< Set-Cookie:
JSESSIONID=LtfrSGfTcdQ3jTyD9pYk5s0lLKrSWf5QLvnK8HplNp8z0rMSzCs2!-1705297486;
path=/
< X-Powered-By: Servlet/2.4 JSP/2.0
< Set-Cookie: BIGipServerpl-prod_iplanet_http_v4=214476992.20480.0000;
path=/
< Vary: Accept-Encoding
< Content-Encoding: gzip
< Content-Length: 31811
< Connection: Keep-Alive

It seems to be working good until now, only the cookies from google
analytic are missing.

Now is executed the following code

$POST_headers = array(
    'Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language: en-US,en;q=0.8',
    'Cache-Control: max-age=0',
    'Connection: keep-alive',
    'Content-Length: 276',
    //'Content-Type: application/x-www-form-urlencoded',
    'Content-Type: multipart/form-data',
    'Cookie: __CT_Data=gpv=1&apv_7197_www07=1;
WRUID=0;'.$m[1][0].';'.$m[1][1].';',
    'Host: www.ura.gov.sg',
    'Origin: https://www.ura.gov.sg',
    'Referer:
https://www.ura.gov.sg/realEstateIIWeb/resiRental/search.action',
    'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML,
like Gecko) Ubuntu Chromium/28.0.1500.71 Chrome/28.0.1500.71 Safari/537.36'
);

$str_postdata="submissionType=pd&from_Date_Prj=JUN-2013&to_Date_Prj=JUN-2013&__multiselect_projectNameList=&__multiselect_selectedProjects=&propertyType=ac&from_Date=MAY-2013&to_Date=JUN-2013&__multiselect_postalDistrictList=&selectedPostalDistricts=01&selectedPostalDistricts=02&selectedPostalDistricts=03&__multiselect_selectedPostalDistricts=";

$arr_postdata = array (
    'submissionType' => 'pd',
    'from_Date_Prj' => 'JUN-2013',
    'to_Date_Prj' => 'JUN-2013',
    '__multiselect_projectNameList' => '',
    '__multiselect_selectedProjects' => '',
    'propertyType' => 'lp',
    'from_Date' => 'MAR-2013',
    'to_Date' => 'JUN-2013',
    '__multiselect_postalDistrictList' => '',
    'selectedPostalDistricts' => '19',
    '__multiselect_selectedPostalDistricts' => ''

    );

curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_ENCODING, "gzip,deflate,sdch");
curl_setopt($ch, CURLOPT_URL, "
https://www.ura.gov.sg/realEstateIIWeb/resiRental/submitSearch.action");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $arr_postdata);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, $POST_headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);

echo curl_exec($ch);

*Request Headers *

POST /realEstateIIWeb/resiRental/submitSearch.action HTTP/1.1
Accept-Encoding: gzip,deflate,sdch
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.8
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 276
Cookie:
JSESSIONID=GJQMSLBQJknnvyvrsBpNkZlgTxLjjB1zT4XJ8zBjFCz53vSgxnrR!806951883;BIGipServerpl-prod_iplanet_http_v4=214476992.20480.0000;
Host: www.ura.gov.sg
Origin: https://www.ura.gov.sg
Referer: https://www.ura.gov.sg/realEstateIIWeb/resiRental/search.action
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like
Gecko) Ubuntu Chromium/28.0.1500.71 Chrome/28.0.1500.71 Safari/537.36
Content-Length: 1280
Expect: 100-continue
Content-Type: multipart/form-data;
boundary=----------------------------5fcfd20988c4

*Request Response*
< HTTP/1.1 400 Bad request
< Date: Sat, 10 Aug 2013 11:05:55 GMT
< Content-length: 267
< Content-type: text/html
< Connection: close

Request is not working properly (400 Bad request).

Only If I use the $str_postdata and 'Content-Type:
application/x-www-form-urlencoded' (with urlencode() function), I can get a
successful response but the form is still not submitted successfully.

* *
*## Process using cURL PHP *(with string for the parameters)*
*

*Request Headers *

POST /realEstateIIWeb/resiRental/submitSearch.action HTTP/1.1
Accept-Encoding: gzip,deflate,sdch
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.8
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 276
Content-Type: application/x-www-form-urlencoded
Cookie:
JSESSIONID=WhbsSLQF20np4qbnl1TbXdWXmbv1MT5g92vwchQphv142DcsvRxk!806951883;BIGipServerpl-prod_iplanet_http_v4=214476992.20480.0000;
Host: www.ura.gov.sg
Origin: https://www.ura.gov.sg
Referer: https://www.ura.gov.sg/realEstateIIWeb/resiRental/search.action
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like
Gecko) Ubuntu Chromium/28.0.1500.71 Chrome/28.0.1500.71 Safari/537.36

*Request Response*

< HTTP/1.1 200 OK
< Date: Mon, 12 Aug 2013 01:49:22 GMT
< Content-length: 1791
< Content-type: text/html;charset=UTF-8
< X-wily-servlet: Encrypt1
WSry9LDfbZEeaCzdlYhs5ZVGfvNRF+pLLMu9RzcTB5ahtwuSbo0msWybnDTnQk5ZkYb9ckzGENTW6lIEDCgt6lqePNgpd97zx8P6moDyz/IGVpmwURz8p3REnaBPhtymMqZTHiw8yf/O+cbDOHIjxi8alNIzlrEeUZN3FabdHm7Eu6rzJPATjImQueUCZDWgKCqz7fpSRy4mupGhwPZG6A==
< X-Powered-By: Servlet/2.4 JSP/2.0
< X-wily-info: Clear guid=0
< Vary: Accept-Encoding
< Content-Encoding: gzip
< Connection: Keep-Alive

But still the form is not working properly and I’m not getting any records
of data back.

Some help will be really appreciated.

Thanks!

Oscar

On 8 August 2013 16:32, Waruna Geekiyanage <waruna.ns_at_gmail.com> wrote:

> ##
> Try changing $str_postdata to an array where field name as key and field
> data as value.
> ie.
> $arr_postdata["submissionType"] = "pd";
> $arr_postdata["from_Date_Prj"] = "JUN-2013";
> .....
> .....
> etc...
>
> And remove header 'Content-Type: application/x-www-form-urlencoded'.
> ##
> Compare the cURL request headers with browser request headers.
> ##
> Make sure the /tmp/cookie contains the cookies.
>
> Thank You,
> Waruna Geekiyanage
>
> M: +94 71 4168164
> E: waruna.ns_at_gmail.com <warunans_at_gmail.com>
> MSN: warunans_at_hotmail.com
> GTalk: waruna.ns
> ICQ: 225184383
> Yahoo: warunans
> Skype: warunans
> On 8/7/2013 4:06 PM, Oscar Vidal wrote:
>
> Hi Waruna!
>
> thanks for you answer.
>
> Here is the code that I'm using currently:
>
> --------------------------------------------------
> $cookieFile="/tmp/cookie";
>
> $ch = curl_init();
> curl_setopt($ch, CURLOPT_COOKIEFILE, $cookieFile);
> curl_setopt($ch, CURLOPT_COOKIEJAR, $cookieFile);
> curl_setopt($ch, CURLOPT_ENCODING, "gzip");
> curl_setopt($ch, CURLOPT_URL, "http://www.some.url/submitSearch.action");
> curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type:
> application/x-www-form-urlencoded'));
>
> // The string is exactly the same that the one generated by the browser
>
>
> $str_postdata="submissionType=pd&from_Date_Prj=JUN-2013&to_Date_Prj=JUN-2013&__multiselect_projectNameList=&__multiselect_selectedProjects=&propertyType=ac&from_Date=MAY-2013&to_Date=JUN-2013&__multiselect_postalDistrictList=&selectedPostalDistricts=01&selectedPostalDistricts=02&selectedPostalDistricts=03&__multiselect_selectedPostalDistricts=";
>
> curl_setopt($ch, CURLOPT_POST, 1);
> curl_setopt($ch, CURLOPT_POSTFIELDS, $str_postdata);
> curl_setopt($ch, CURLINFO_HEADER_OUT, 1);
> curl_setopt($ch, CURLOPT_VERBOSE, 1);
>
> curl_exec($ch);
>
> $info = curl_getinfo($ch, CURLINFO_HEADER_OUT);
>
> echo $info;
>
> curl_close($ch);
> --------------------------------------------------
>
> The content of $info is
>
>
> POST /submitSearch.action HTTP/1.1
> Host: www.ura.gov.sg
> Accept: */*
> Accept-Encoding: gzip
> Cookie: BIGipServerpl-prod_iplanet_http_v4=214476992.20480.0000;
> JSESSIONID=vjyWSCpX2G9V1tKpTTHxcZh0t9nTyyqtjC291JrtXJC1vk70m0xm!-1705297486
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 330
>
>
> Hopefully it can give some clue about why the post to the form is not
> working.
> If I can add some other valuable info, please just ask me :)
>
> Thanks in advance!!
> Oscar
>
>
>
> On 7 August 2013 17:44, Waruna Geekiyanage <waruna.ns_at_gmail.com> wrote:
>
>> submit your PHP/cURL code, so that we can give a clear answer.
>>
>> Use following settings to submit POST data
>> curl_setopt ($ch, CURLOPT_POST, 1);
>> curl_setopt ($ch, CURLOPT_POSTFIELDS, $arr_postdata);
>>
>> $arr_postdata - field name as key and field data as value
>>
>> use following setting for debugging
>>
>> curl_setopt($ch, CURLINFO_HEADER_OUT, 1);
>> $info = curl_getinfo($ch, CURLINFO_HEADER_OUT);
>>
>> $info will give the request headers.
>>
>> Thank You,
>> Waruna Geekiyanage
>>
>> M: +94 71 4168164 <%2B94%2071%204168164>
>> E: waruna.ns_at_gmail.com <warunans_at_gmail.com>
>> MSN: warunans_at_hotmail.com
>> GTalk: waruna.ns
>> ICQ: 225184383
>> Yahoo: warunans
>> Skype: warunans
>> On 8/7/2013 2:32 PM, Oscar Vidal wrote:
>>
>> Hi!
>>
>> I'm trying to submit a (java servlet) form using CURL in PHP, but it
>> seems like there is a problem with the parameters. I cant really understand
>> why its happening since I'm testing the CURL with a identical string
>> parameters that is being used by the browser.
>>
>> After some research in diverse forums I wasn't able to find a solution to
>> my particular problem.
>>
>> this is the POSTFIELDS string generated by the browser (and working):
>>
>> submissionType=pd&__multiselect_PostCodeList=&selectedPostCode=01&selectedPostCode=02&selectedPostCode=03&__multiselect_selectedPostCodes=
>>
>> and I'm using and identical (for testing) string in the PHP script but
>> it im getting a HTML file as a answers telling "Missing parameters in
>> search query".
>>
>> I believe that the form
>>
>> __multiselect_PostCodeList=
>> &selectedPostCode=01
>> &selectedPostCode=02
>> &selectedPostCode=03
>> &__multiselect_selectedPostCodes=
>>
>> is quite wired (never see before this) and I'm wondering that it can be
>> the reason of why the post is not working from CURL.
>>
>> The form seems to be successfully submitted since I'm getting this header
>>
>> HTTP/1.1 200 OK
>> Date: Wed, 07 Aug 2013 08:02:56 GMT
>> Content-length: 1791
>> Content-type: text/html;charset=UTF-8
>> X-Powered-By: Servlet/2.4 JSP/2.0
>> Vary: Accept-Encoding
>> Content-Encoding: gzip
>> Connection: Keep-Alive
>>
>> Note: I tried submitting the same form from *Lynx* and I'm also getting
>> the same result ("Missing parameters in search query"). So it seems like
>> its only working with browsers like Mozilla or Chrome.
>>
>> Please some help will be really appreciated, I don't have any more ideas
>> at this point.
>>
>> Thanks! Oscar
>>
>>
>> _______________________________________________http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
>>
>>
>>
>
>

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Received on 2013-08-12