Hi

I have set the max-connect ("-m") option to 15 seconds, and yet some
websites don't return the pages in that much time. These pages are
definitely valid if I visit them from within a browser, they are not
404 or anything else.

After some investigation, I am discovering that many websites can in
fact tell that their page was requested through a Curl call. Is this
something that we can do nothing about, or can we request pages in a
manner that masks the request call as being from a regular browser,
and does not sound like a Curl request at all?

Thanks for any pointers or advice. I am hoping to get some ideas
beyond "Just reveal your identity as it is", because that doesn't work
either.

EP
Received on 2007-06-15