Hi, David
For the 'robot', I think you may try to pretend to be a human by using User Agent, for example in PHP.
------------------------------------------------------------------------------------$curl = curl_init();curl_setopt($curl,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');------------------------------------------------------------------------------------
This code will pretend to be a person who is using Firefox on Windows Machine.
Regards,Ethan

From: dolan2go_at_yahoo.com
Subject: Re: Curl and recurring timeouts
Date: Sat, 1 Sep 2012 11:41:44 -0400
To: curl-and-php_at_cool.haxx.se

Thomas,
How can that be explained? What might be the cause of all these timeouts?
Do these load times occur from within a browser? It could be caused by many things. Need more info.
As a side note, I recently used a cURL script to obtain info from a google search page. They were detecting what was perceived as a 'robot' and denying the page, until verified by a human. It came and went. The threshold for displaying could not be determined. They might have many 'detectors' on their sites.
How is cUrl different than an access from a browser?
Can't answer this one.
Have you tried increasing the timeout?
David
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Received on 2012-09-02