I'm not a curl expert, but I thought I knew what I was doing ! Source is at
end of this post.
I am trying to make my app look like a browser to get a http request to an
outside site (sending SMS messages). App is in php on a local server.
Problem is local network is within an education authority, and all access
to internet is thru their proxy server (filters and does all sorts of
stuff.)

The proxy is a bit weird, so I've been using Fiddler to debug the traffic
from a browser and from my app to see difference (because the proxy is
blocking me with 403 Forbidden, so I'm guessing I'm not being a browser
correctly).

But: when using Fiddler(to debug the HTTP conversation), there is a
difference in the conversation between my app and the target site, and a
browser and the target site.

Picture is like this:

Browser <--->Fiddler Proxying 127.0.0.1:8888 <---Real proxy:80 --->
www.google.co.uk
My APP <--->Fiddler Proxying 127.0.0.1:8888 <---Real proxy:80 --->
www.google.co.uk

Browser conversation goes like this:
------------------------------------
Connection 1:
GET / HTTP/1.1
Host: www.google.co.uk
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9)
Gecko/2008052906 Firefox/3.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Proxy-Connection: keep-alive
Cookie:
PREF=ID=09d7d0513a2af2a7:TM=1214992131:LM=1214992131:S=AS1YPHde9fVL6i3D;
NID=12=B98NJ2rNDeUzjBxixRIbSVGjIZGXSFbG_llKPHF3FnwyB5uSJqBgVkwFCRw3nk1JvpV_q37nfFe960az37YZabzJQ_jBjR4fg9bLm8x4U0XJK9flYY00b2wMMnxSieki

receives 302 redirect :
HTTP/1.0 302 Moved Temporarily
Server: squid/2.5.STABLE12
Date: Wed, 02 Jul 2008 11:48:01 GMT
Content-Length: 0
Location:
http://securelogin.leedslearning.net/dpid=1&cat=101&ttl=0&groupname=-&policyname=-&username=-&userip=10.6.3.20&connectionip=10.253.21.61&nsphostname=lln-civ-nsp-02.leedslearning.net&protocol=squid25&dplanguage=-&url=aHR0cDovL3d3dy5nb29nbGUuY28udWsv

then a number of redirects to https, and then a proxy login page (which is
what I was hping to get)

My App conversation goes like this:
-----------------------------------
CONNECT www.google.co.uk:80 HTTP/1.0
Host: www.google.co.uk:80
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;
InfoPath.1; .NET CLR 2.0.50727)
Accept: */*
Accept-Language: en-gb
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Cookie:
PREF=ID=7c5d00a43b1c1c23:TM=1214992129:LM=1214992129:S=Xtw0zCpIwJa-MjnA

note the CONNECT request through tunnel, rather than the GET as with the
browser.

The curl error is "56 Received HTTP code 403 from proxy after CONNECT"

and the result of the verbose STDERR logging is
* About to connect() to proxy 127.0.0.1 port 8888 (#0)
* Trying 127.0.0.1... * connected
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
* Establish HTTP proxy tunnel to www.google.co.uk:80
> CONNECT www.google.co.uk:80 HTTP/1.0
Host: www.google.co.uk:80
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;
InfoPath.1; .NET CLR 2.0.50727)
Accept: */*
Accept-Language: en-gb
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Cookie:
PREF=ID=7c5d00a43b1c1c23:TM=1214992129:LM=1214992129:S=Xtw0zCpIwJa-MjnA

< HTTP/1.0 403 Forbidden
< Server: squid/2.5.STABLE12
< Mime-Version: 1.0
< Date: Wed, 02 Jul 2008 11:50:33 GMT
< Content-Type: text/html
< Content-Length: 1096
< Expires: Wed, 02 Jul 2008 11:50:33 GMT
< X-Squid-Error: ERR_ACCESS_DENIED 0
< X-Cache: MISS from lln-spp-nc-03.leedslearning.net
< Proxy-Connection: keep-alive
<
* Received HTTP code 403 from proxy after CONNECT
* Closing connection #0

Why is my app doing a CONNECT, and the browser doing a GET www.google.co.uk
?
How can I set up curl to emulate the browser conversation ?

Note - it's not a browser (IE) funny - fiddler binds into IE, but the
browser conversation above is Firefox, installed clean on the machine and
proxy set to 127.0.0.1:8888

Hope someone can shed some light on this, I'm pulling my hair out.
David

Source code for php test is :
$fp=fopen('stderr.txt', 'w');
$headers = array(
                "Accept: */*",
                "Accept-Language: en-gb",
                "Accept-Encoding: gzip, deflate",
                "Proxy-Connection: Keep-Alive",
                "Cookie:
PREF=ID=7c5d00a43b1c1c23:TM=1214992129:LM=1214992129:S=Xtw0zCpIwJa-MjnA",
                );
$agent="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1;
.NET CLR 2.0.50727)";
$ch=curl_init();
$r=curl_setopt($ch, CURLOPT_URL, "http://www.google.co.uk");
$r=curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$r=curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$r=curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$r=curl_setopt($ch, CURLOPT_USERAGENT, $agent);
$r=curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

$r=curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
$r=curl_setopt($ch, CURLOPT_PROXYPORT, 8888);
$r=curl_setopt($ch, CURLOPT_PROXY, '127.0.0.1');

$r=curl_setopt($ch, CURLOPT_VERBOSE, 1);
$r=curl_setopt($ch, CURLOPT_STDERR, $fp);

$data=curl_exec($ch);
$e=curl_errno($ch).' '.curl_error($ch);

curl_close($ch);

fclose($fp);

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Received on 2008-07-02