curl-and-php
Thank you for your answer...
Date: Fri, 07 Dec 2007 15:37:02 -0800
Hi Morgan,
Thank you for your answer. I understand that the browser makes
different headers for different kinds of files and that it will send
some MIME types to certain applications etc. The fundemental problem is
that the csv would be downloading sort of as a side effect and I am not
capturing it at all. When I do livehttpheaders it just sends one for a
normal html page. When you press the download button manually it
doesn't even trigger a page refresh So I don't know where the
download is going but it is not coming in on the curl exec when i try to
make that copy what the submit button is doing.
This is the latest rev of my code. I put it up here a couple of weeks
ago. but I've fixed a couple of typos since. It was crashing java
(which is what the target app is written in) because I had a typo in one
of the post vars.
What gets output to the output file is "1" or nothing... just exactly
what was echoing when I was just echoing it to the browser. There
should be transactions data. I didn't seem to need cookies, I added
cookiefile and cookiejar as per your suggestion but it didn't help. I
also didn't seem to need the CAfile, I am logging in fine without it.
//custom code to check if user is logged in omitted
$fname .= 'GatewayTxns' . date("F-j-Y",time()) . '.csv'; //create file
with today's date,
$path = '/path/to/it'; //config
$fname = $path . FSDLM . $fname; //FSDLM is the File System
Delimiter / for unix systems
//$fp = fopen($fname,'w');
$cookiesfile = "/tmp/Cookies";
$login_data = array(
'storename' => 'xxxxxxxx',
'&username' => 'xxxxxxxx',
'&passwd' => 'xxxxxx'
);
$data =
curl_It_Here('https://www.linkpointcentral.com/lpc/servlet/LPCLogin',encode_data($login_data));
/*Example validation stuff we have to screen scrape this :P
st="3:0EC1389E5941C2D2651CE34A1EED1FCA";
si="3:384BD4235FCC6D02384BD4235FCC6D020EC1389E5941C2D22C28489A3D962314";
sk="3:9FA84772CEA04892627C90399391521F";
*/
$st=scrape($data,"st=",4,'";');
$si=scrape($data,"si=",4,'";');
$sk=scrape($data,"sk=",4,'";');
//echo "st=$st si=$si sk=$sk"; exit;
$url = "https://www.linkpointcentral.com/lpc/servlet/LPCReport";
//now curl directly to the transaction report...
//we need to do this intermediate step so that Linkpoint will tell us
the temp file name
//set up the post fields
$postdata = array(
'_st' => $st ,
'_si' => $si ,
'_sk'=> $sk ,
'allowcredit'=>'1' ,
'allowtxn'=>'1' ,
'rptaction'=>'qryordertime' ,
'period'=>'yesterday', //for testing
'approved'=>'1' , //I want to only see approved
orders, ur results may vary
'pagesize'=>'100'
);
$data = curl_It_Here($url,encode_data($postdata));
//now we need to scrape the total the pagefile name and the grand
total at least out of here...
$pagefile=scrape($data,'pagefile.value = "', 18, '";') ;
$grandtotal=scrape($data,'"grandtotal" value="',20, '">');
$numrecords=scrape($data, '"numrecords" value="', 20, '">');
//let's suck up a few more things
$timeperiod=scrape($data,'name="timeperiod" value="', 25,'">');
//maybe it just likes to have these...
$fromday = scrape($data,'name="fromday" value="',22,'">');
$frommonth = scrape($data,'name="frommonth" value="',24,'">');
$fromyear= scrape($data,'name="fromyear" value="',23,'">');
$today = scrape($data,'name="today" value="',20,'">');
$tomonth = scrape($data,'name="tomonth" value="',22,'">');
$toyear = scrape($data,'name="toyear" value="',21,'">');
//echo ("$fromday/$frommonth/$fromyear $today/$tomonth/$toyear");
exit;
//lets rescrape those wretched variables maybe they changed
$st=scrape($data,"st=",4,'";');
$si=scrape($data,"si=",4,'";');
$sk=scrape($data,"sk=",4,'";');
//echo "st=$st si=$si sk=$sk"; exit;
//echo (" $pagefile $grandtotal $numrecords $timeperiod" ); exit;
$rptpostvars = array(
'pageno' => '1' ,
'pagefile' => $pagefile ,
'pagesize' => '100' ,
'numrecords' => $numrecords ,
'grandtotal' => $grandtotal ,
'start' => '0' ,
'rptaction' => 'qryordertime' ,
'period' => 'yesterday' , //for testing
'timeperiod' => $timeperiod ,
'fromday' => $fromday ,
'frommonth' => $frommonth ,
'fromyear' => $fromyear ,
'today' => $today ,
'tomonth' => $tomonth ,
'toyear' => $toyear ,
'lastx' => '',
'expflag' => '1' ,
'exptype' => 'csv' ,
'_st' => $st ,
'_si' => $si ,
'_sk' => $sk ,
'storename' => '' ,
'timeperiod' => "$frommonth/$fromday/$fromyear -
$tomonth/$today/$toyear",
'userid' => '' ,
'oid' => '' ,
'startparent' => '',
'allowcredit' => '1' ,
'allowtxn' => '1' ,
'expfile' => 'exp_ordercsv.xsl' ,
'target' => '', //lets set after '_self'
) ;
$data = curl_It_Here($url,encode_data($rptpostvars) ); //url encode
vars with : in their value it breaks it
if (file_put_contents($fname, $data) === false) {
print('Failed to write file contents to '.$fname."\n");
}
exit;
function curl_It_Here($url,$postvars,$file=NULL,$debug=NULL) {
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
//curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postvars);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiesFile);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiesFile);
if ($file) {
curl_setopt($ch, CURLOPT_FILE, $file);
}
$data = curl_exec($ch);
if (!is_null($debug)) {
$e = curl_error($ch);
print_r($e);
$f= curl_getinfo($ch);
print_r($f);
exit;
}
curl_close($ch);
unset($ch);
return $data;
}
function encode_data($data)
{
$a ='';
foreach ($data as $k=>$v) {
$a .= $k . '=' . htmlentities(urlencode($v)) . '&';
}
return substr($a,0,-1);
}
function scrape($data, $startstr, $offset, $endstr) {
$start =strpos($data,$startstr);
$rest=substr($data,$start+$offset);
$end=strpos($rest,$endstr);
return substr($rest,0,$end);
//return ($answer);
}
On Fri, 2007-12-07 at 10:34 -0800, Morgan Galpin wrote:
> Oops,
>
> After I sent my last message I noticed a problem with my code. I wasn't
> properly encoding the post data. The fixed version is below.
>
> <?php
>
> // Set parameters.
> $url = "https://www.linkpointcentral.com/lpc/servlet/LPCReport";
> $certificateFile = 'C:/php/curl-ca-bundle.crt';
> $cookiesFile = 'C:/windows/temp/cookies.txt';
> $outputFilename = 'C:/output_file.xsl';
>
> // Set up the post data.
> $postData = array(
> 'pageno' => '1',
> 'expflag' => '1',
> 'rptaction' => 'qryordertime',
> // ... etc.
> );
> $encodedPostData = array();
> foreach ($postData as $key => $value) {
> $encodedPostData[] = $key.'='.htmlentities(urlencode($value));
> }
> $postFields = implode('&', $encodedPostData);
> print('Post fields: '.$postFields."\n");
>
> // Create the curl handle with the data to send.
> $ch = curl_init();
> curl_setopt($ch, CURLOPT_POST, true);
> curl_setopt($ch, CURLOPT_POSTFIELDS, $postFields);
> curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
> curl_setopt($ch, CURLOPT_URL, $url);
>
> // Set other options.
> curl_setopt($ch, CURLOPT_VERBOSE, true);
> curl_setopt($ch, CURLOPT_CAINFO, $certificateFile);
> curl_setopt($ch, CURLOPT_CAPATH, $certificateFile);
> curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
> curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
> curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiesFile);
> curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiesFile);
>
> // Get the result and output.
> $result = curl_exec($ch);
>
> // Output the result to a file.
> if (file_put_contents($outputFilename, $result) === false) {
> print('Failed to write file contents to '.$outputFilename."\n");
> }
>
> // Close the handle.
> curl_close($ch);
>
> ?>
>
> I hope *that* helps. :-)
>
> Morgan Galpin.
>
> Morgan Galpin wrote:
> > Hi Colleen,
> >
> > I think what might be happening with the files you're trying to download
> > is that the server is setting certain http header values to indicate to
> > your web browser that it should download the file rather than display
> > the file. Every link that you click on in a web browser simply downloads
> > the file that is linked to. The browser decides whether or not to
> > display the file or to ask the user to save it somewhere. Browsers know
> > how to display html, text, and images, which is why they never ask about
> > those. When the server wants to force the client browser to save the
> > file rather than display it, because the browser will try to display an
> > xsl file, it will include a header like, 'Content-Disposition:
> > attachment; filename="somefile.xsl";'. When the client browser receives
> > a file with a header like this, it will normally display a "Save As"
> > dialog where the user can choose to save the file.
> >
> > So, there should be no difference for trying to save the file received
> > in this manner with curl versus any other URL. I don't know what your
> > code looks like, but I would try something like the following:
> >
> > <?php
> >
> > // Set parameters.
> > $url = "https://www.linkpointcentral.com/lpc/servlet/LPCReport";
> > $certificateFile = 'C:/php/curl-ca-bundle.crt';
> > $cookiesFile = 'C:/windows/temp/cookies.txt';
> > $outputFilename = 'C:/output_file.xsl';
> >
> > // Set up the post data.
> > $postData = array(
> > 'pageno' => '1',
> > 'expflag' => '1',
> > 'rptaction' => 'qryordertime',
> > // ... etc.
> > );
> > $postFields = implode('&', $postData);
> >
> > // Create the curl handle with the data to send.
> > $ch = curl_init();
> > curl_setopt($ch, CURLOPT_POST, true);
> > curl_setopt($ch, CURLOPT_POSTFIELDS, $postFields);
> > curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
> > curl_setopt($ch, CURLOPT_URL, $url);
> >
> > // Set other options.
> > curl_setopt($ch, CURLOPT_VERBOSE, true);
> > curl_setopt($ch, CURLOPT_CAINFO, $certificateFile);
> > curl_setopt($ch, CURLOPT_CAPATH, $certificateFile);
> > curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
> > curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
> > curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiesFile);
> > curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiesFile);
> >
> > // Get the result and output.
> > $result = curl_exec($ch);
> >
> > // Output the result to a file.
> > if (file_put_contents($outputFilename, $result) === false) {
> > print('Failed to write file contents to '.$outputFilename."\n");
> > }
> >
> > // Close the handle.
> > curl_close($ch);
> >
> > ?>
> >
> > Let us know if that helps,
> > Morgan Galpin.
> >
> >
> > Colleen R. Dick wrote:
> >
> >> I have managed to log into a site and make it do a query and have my
> >> custom data appear on a dynamic web page (if I choose to print out the
> >> results of curl_exec.)
> >> There is a BUTTON on that page that allows you to download a spreadsheet
> >> of that data. You can't just fetch the file with wget or curl because
> >> I'm sure it doesn't live anywhere under document root so it wouldn't
> >> have a URL. you have to submit the form to ask the server for it and it
> >> downloads it as some sort of attachment with a dialog when you do it
> >> manually. Obviously I want to set it up so it just falls into some
> >> file handle. I checked what the button does, it just calls a
> >> javascript to set up some other post variables before submitting the
> >> form.
> >>
> >> Here is the submit button:
> >>
> >> <input class="button" type="button" value="Export All Data"
> >> onclick="exportFile(); return(false);
> >>
> >> Here is the javascript exportFile:
> >>
> >> function exportFile()
> >> {
> >> document.RptForm.pageno.value = 1;
> >> document.RptForm.expflag.value = 1;
> >> document.RptForm.rptaction.value = "qryordertime";
> >>
> >> document.RptForm.pagefile.value = "temp1196982492309.1040408.1";
> >>
> >> exptype = document.RptForm.exptype.options[document.RptForm.exptype.selectedIndex].value;
> >> if (exptype=="csv") document.RptForm.expfile.value="exp_ordercsv.xsl";
> >> else if(exptype=="xml") {
> >> document.RptForm.target = "_blank";
> >> document.RptForm.expfile.value="exp_orderxml.xsl";
> >> }
> >> document.RptForm.submit();
> >> document.RptForm.target = "_self";
> >> }
> >>
> >> So what I did was hard code all the post variables like the javascript does them
> >> and submits the form.
> >> It doesn't throw any errors, I don't think. I made it print out curl_getinfo
> >> Array
> >> (
> >> [url] => https://www.linkpointcentral.com/lpc/servlet/LPCReport
> >> [content_type] => text/html; charset=ISO-2259-1
> >> [http_code] => 200
> >> [header_size] => 153
> >> [request_size] => 665
> >> [filetime] => -1
> >> [ssl_verify_result] => 0
> >> [redirect_count] => 0
> >> [total_time] => 2.981608
> >> [namelookup_time] => 0.000101
> >> [connect_time] => 0.113942
> >> [pretransfer_time] => 0.529531
> >> [size_upload] => 0
> >> [size_download] => 0
> >> [speed_download] => 0
> >> [speed_upload] => 0
> >> [download_content_length] => 0
> >> [upload_content_length] => 0
> >> [starttransfer_time] => 2.981528
> >> [redirect_time] => 0
> >> )
> >>
> >> It names the document rpt*******.csv but I would like it to download into a file name of my choice.
> >> To that end I have set up and opened a filehandle but it always ends up of size zero, which makes sense if
> >> I don't know how to tell CURL to use it. Where is the document going if anywhere?
> >> It is not in tmp or in my web tree. I'm sorry if I am stupid but I have really looked very hard for the answer.
> >> All I see is little examples to use Curl to download a file where it has a URL. I know how to do that.
> >>
> >> _______________________________________________
> >> http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
> >>
> >>
> > _______________________________________________
> > http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
> >
> _______________________________________________
> http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
>
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Received on 2007-12-08