Are the requests you are making, to static files, or are they to
dynamically generated resources?

And it's certainly possible to obtain a 404 response from a site for
something that should not be missing: all sorts of wacky internal
implementations of sites can lead to this as can self-defence measures
such as your suggestion that it's detecting too frequent a pattern of hits.

If you try these requests in your browser, do you ever see failed
requests there, too? Can you isolate whether it's cURL only? How about
if you repeatedly refresh the URL in the browser?

Yahoo might be seeing a heavy load and rejecting you for it; you may
indeed want to put a sleep() in between requests.
Received on 2007-01-29