Re: working around status code 429

From: Dirk Hünniger via curl-users <curl-users_at_lists.haxx.se>
Date: Wed, 31 Jan 2024 13:08:58 +0100

Hi Ray,

sorry for the delay. I updated the output file in the file curlrun1 to
point to tmp, which is a path that exists. I attached a 7zip file
"files.7z" of the input file "curlrun1" and the output file "out".  I
used the following curl command:

curl --retry 30 -v -w "%{http_code} %{url} %{filename_effective}\n"
--compressed -K ./curlrun1 --parallel -L > out 2>&1

When I do

cat out | grep "429 h"

I get the output:

429 https://de.wikibooks.org/File:Parathion-skeletal.png /tmp/626.2.html
429 https://de.wikibooks.org/wiki/Wikibooks:Sammlungen/File:Betain.svg
/tmp/637.3.html
and more.

So to my understanding the are 429 errors, which are not retried,
although I used --retry in the call to curl.

Yours Dirk

On 26.01.24 08:28, Ray Satiro via curl-users wrote:
> On 1/25/2024 5:10 AM, Dirk Hünniger via curl-users wrote:
>>
>> I attached the output of
>>
>> curl --retry 30 -v -w "%{http_code} %{url} %{filename_effective}\n"
>> --compressed -K ./curlrun1 --parallel -L > out 2>&1
>>
>> but the file is so large that I cannot really work with well it in
>> gedit.
>>
>
> The output shows that the transfers that return HTTP 429 have failed
> with curl error CURLE_WRITE_ERROR (23) and that is at least one reason
> why they are not being retried. You can try with the latest version of
> curl but I suspect this is not going to be a curl issue. Also, if
> gedit isn't working for you then use something else. There's no reason
> you shouldn't be able to work with large files like that in a text
> editor.
>
> Here's one example. They all look the same:
>
> Warning: Failed to create the file
> Warning: /tmp/MediaWiki2LaTeXImages-3943f5a7379c6219/827.4.html: No
> such file
> Warning: or directory
> * Failure writing output to destination
> * stopped the pause stream!
> curl: (23) Failure writing output to destination
>
>

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2024-01-31