On Mon, 17 Jan 2005, IG wrote:

> thats the exact message displyed after fetching the above mentioned page.
> "Your web-client is unable to display this resource. Don't use content
> grabbers here. Pay some respect to my work ..."

Use LiveHTTPHeaders and make your requests look as similar to those as
possible.

But I strongly urge you to acknowledge the wishes of the authors. If they
don't want you to "robot" the contents, then you should probably not do it.

> So despite the fact that user agent is properly set- curl is still
> recognized as a bot. I think theres something about javascript on the page
> but at the same time curl understands javascript right? Any thoughts?

curl does NOT support javascript so it would then rather be the lack of
javascript support that makes it appear as a robot. Which of course would be a
silly way to detect that...

-- 
      Daniel Stenberg -- http://curl.haxx.se -- http://daniel.haxx.se
       Dedicated custom curl help for hire: http://haxx.se/curl.html