Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: curl: add --rate to set max request rate per time unit
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Daniel Stenberg via curl-users <curl-users_at_lists.haxx.se>
Date: Mon, 4 Apr 2022 16:01:18 +0200 (CEST)
On Mon, 4 Apr 2022, Timothe Litt via curl-users wrote:
>> Allow 15 transfers per second:
>>
>> curl --rate 15/s $URL[a-z] -O
> I suppose this is to outwit hosts that have a corresponding limit in their
> server(s). I don't recall frequent use of transfer rate limits, though
> connection rate limits are more common.
> (And easy to implement in firewalls
> -- e.g. iptables -m{limit,connlimit,hashlimit}.) The distinction matters
> when connections are reused, which curl tries to do...one connection can
> serve many transfers.
The difference between those two I believe isn't going to be too important for
this.
There are lots of HTTP API endpoints that implement transfer rate limits. As
in they only accept N requests per M time units. There's even this ongoing
effort to make standardized HTTP response headers with the rate-limiting
values:
https://datatracker.ietf.org/doc/html/draft-polli-ratelimit-headers
(and we could possibly make curl adapt to those values in a future)
> Specifying (in the man page) how it interacts with --limit-rate,
> --speed-limit, --parallel seems necessary for users to understand what
> happens.
They're all independent from each other so I'm not sure what I should add.
Except for --parallel, as I've specifically mentioned that this feature is for
serial (non-parallel) transfers.
> It's not clear what benefit this has to the client machine (the one running
> curl).
To avoid (HTTP) errors that curl otherwise gets when the server rejects the
too fast requests.
> So perhaps this should be applied per server (target/URL authority)
> host?
It could, but I think that would mostly complicate the option for something
most users won't care about.
Date: Mon, 4 Apr 2022 16:01:18 +0200 (CEST)
On Mon, 4 Apr 2022, Timothe Litt via curl-users wrote:
>> Allow 15 transfers per second:
>>
>> curl --rate 15/s $URL[a-z] -O
> I suppose this is to outwit hosts that have a corresponding limit in their
> server(s). I don't recall frequent use of transfer rate limits, though
> connection rate limits are more common.
> (And easy to implement in firewalls
> -- e.g. iptables -m{limit,connlimit,hashlimit}.) The distinction matters
> when connections are reused, which curl tries to do...one connection can
> serve many transfers.
The difference between those two I believe isn't going to be too important for
this.
There are lots of HTTP API endpoints that implement transfer rate limits. As
in they only accept N requests per M time units. There's even this ongoing
effort to make standardized HTTP response headers with the rate-limiting
values:
https://datatracker.ietf.org/doc/html/draft-polli-ratelimit-headers
(and we could possibly make curl adapt to those values in a future)
> Specifying (in the man page) how it interacts with --limit-rate,
> --speed-limit, --parallel seems necessary for users to understand what
> happens.
They're all independent from each other so I'm not sure what I should add.
Except for --parallel, as I've specifically mentioned that this feature is for
serial (non-parallel) transfers.
> It's not clear what benefit this has to the client machine (the one running
> curl).
To avoid (HTTP) errors that curl otherwise gets when the server rejects the
too fast requests.
> So perhaps this should be applied per server (target/URL authority)
> host?
It could, but I think that would mostly complicate the option for something
most users won't care about.
-- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-users Etiquette: https://curl.haxx.se/mail/etiquette.htmlReceived on 2022-04-04