I don't know how they store their database data. As a user, I can use
cURL or my browser to request the daily, weekly or monthly data
between a beginning and ending date that I specify. I have never seen
it fail using the browser. On Yahoo finance site, a user can read the
data in html tabular format or convert it directly into csv format.

The tURL entry into the cURL command looks like the following…

ichart.finance.yahoo.com/table.csv?

The tParam entry looks like the following…

s=IBM&a=05&b=10&c=2004&d=05&e=16&f=2006&g=w&ignore.csv

This should retrieve weekly price data for IBM between the dates
6/10/04 through 6/16/06.

For some reason, months have to be entered as 0-11 for Jan though Dec.

As I mentioned in my first message, I know enough to get the command
to work but not much more and it worked flawlessly until the last
month or two. Then I started to encounter the error messages. Today,
I tried adding a random delay into the data request loop so that each
call is delayed between 20 to 300 ticks. It had no noticeable effect.
Repeated calls for the incomplete data will eventually work but it
takes five or six iterations, each iteration generating a percentage
of the missing data.
I'm baffled!

On Jan 28, 2007, at 6:24 PM, Daniel Beardsmore wrote:

> Are the requests you are making, to static files, or are they to
> dynamically generated resources?
>
> And it's certainly possible to obtain a 404 response from a site
> for something that should not be missing: all sorts of wacky
> internal implementations of sites can lead to this as can self-
> defence measures such as your suggestion that it's detecting too
> frequent a pattern of hits.
>
> If you try these requests in your browser, do you ever see failed
> requests there, too? Can you isolate whether it's cURL only? How
> about if you repeatedly refresh the URL in the browser?
>
> Yahoo might be seeing a heavy load and rejecting you for it; you
> may indeed want to put a sleep() in between requests.
Received on 2007-01-29