I am using curl to pull data from other sites' web pages. However, some
sites are not working. curl is using cookies and http authentication
successfully. However, the sites in question return 301 or 302 status codes.
If curl is set to follow the redirection links, it goes to wrong pages, as
the links provided are not correct.

It seems as if the sites are returning 3xx error codes and false links to
prevent access. But how would they know who to block access to? Somehow the
browser gets the right pages. What is different about the request coming
from curl?

The only difference I can find is the IP. I am wondering if they block
access to Class C IP addresses such as my web host has, but not Class A IP
addresses such as my ISP has. Does anyone know if this technique is used? I
thought maybe ISPs usually have Class A addresses while web hosting
providers usually have Class C addresses. Anyone know?

Either that or maybe curl isn't handling it right. Like maybe it's not
receiving the whole response. I doubt that though. Here is an example
response:

HTTP/1.1 302 Found Date: Tue, 12 Mar 2002 23:20:29 GMT Server: Apache/1.3.22
Ben-SSL/1.44 (Unix) AuthMySQL/2.20 PHP/4.1.2 X-Powered-By: PHP/4.1.2
Location: http://www.foo.com/login.html Transfer-Encoding: chunked
Content-Type: text/html

Plus a small html page.

Anyone have any ideas?

Thanks.
Received on 2002-03-13