curl-users
Re: Random error
Date: Sun, 28 Jan 2007 18:12:58 -0600
Daniel, I'm not sure if I'm responding to your previous message
correctly. If not, please direct me how to do so.
Anyway, you could be right but I don't think so. The main reason I
don't think so is that I don't just retrieve data from 1 stock. I
have a script that retrieves data for more than 1000 stocks.
Recently, I got data for 80% of the stocks and errors for the other
20%. I immediately reran the 20% that gave me errors and got data for
80% of those. Each iteration gave me approximately 80% of the
residual data until I got all the data. It seems improbable that this
can be explained by missing files since there was only minutes
between each rerun.
The 80% figure varies but is typical.
There have been times when I get much larger errors. Earlier today
for example, I actually had a situation where I only got data for 1
out of 1000. Immediate rerunning gave me data for 200, another rerun
gave me data for another 600 and subsequent reruns gave me the
remaining data.
If Yahoo was working on the files, I would expect some pattern to the
unavailability of the files but I have encountered the same problem
in the middle of the night when I have run the same procedure.
Knowing this, do you still think it is explained by the file not
being available? Surely they would feed a more appropriate message
than the one I cited in my original message?
Related to your thought, could it be possible that demand for Yahoo
downloads has increased recently and caused the cURL command to time
out before getting the data. If so, is it possible to adjust the wait
time for the command before it times out?
I appreciate the help.
On Jan 28, 2007, at 4:49 PM, Daniel Stenberg wrote:
> On Sun, 28 Jan 2007, Paul Chapman wrote:
>
>> The error message I receive instead of the data is the following?
>>
>> "<!doctype html public "_//W3C//DTD HTML 4.01//EN" "http://
>> www.w3.org/TR/html4/strict.dtd"><html><head><title> Yahoo! - 404
>> Not Found </title><style>/*
>>
>> As I mentioned above, when I resend the correct data will be
>> returned about 80% of the time but about 20% of the time I get the
>> error message.
>
> To me, it seems like quite simply the server says the file doesn't
> exist 20% of the time. Not much for you or us to do about it, other
> than to retry and tell the server admins about their problem.
>
> --
> Commercial curl and libcurl Technical Support: http://haxx.se/
> curl.html
Received on 2007-01-29