Re: curl: add --rate to set max request rate per time unit

From: Timothe Litt <litt_at_acm.org>
Date: Mon, 4 Apr 2022 09:20:13 -0400

On 04-Apr-22 08:42, Daniel Stenberg via curl-users wrote:
> Hi team,
>
> I took a stab at a first implementation of a TODO item from 2019.
>
> When curl is told to do more than one (serial) transfer, we can give
> it max frequency in transfers per time unit. Example:
>
> Allow 12 transfers per minute:
>
>   curl --rate "12/m" $URL $URL
>
> Allow 24 transfers per hour:
>
>   curl --rate "24/h" $URL[0-1000] -O
>
> Allow 15 transfers per second:
>
>   curl --rate 15/s $URL[a-z] -O
>
> https://github.com/curl/curl/pull/8671
>
> Thoughts?
>
I suppose this is to outwit hosts that have a corresponding limit in
their server(s).  I don't recall frequent use of transfer rate limits,
though connection rate limits are more common.  (And easy to implement
in firewalls -- e.g. iptables -m{limit,connlimit,hashlimit}.)  The
distinction matters when connections are reused, which curl tries to
do...one connection can serve many transfers.

Specifying (in the man page) how it interacts with --limit-rate,
--speed-limit, --parallel seems necessary for users to understand what
happens.

It's not clear what benefit this has to the client machine (the one
running curl).  So perhaps this should be applied per server (target/URL
authority) host?  In addition to better matching the likely server
limits, this would allow semantics for use with --parallel to be better
defined.  (The case here is curl x://hosta/a x://hosta/b x://hostb ...)

Depending on the remote limit mechanism, the curl limit might even want
to be per-host per protocol.

It seems to me that the best implementation of client limiting is one
that maximizes client thruput while not exceeding host limits.  A heavy
handed approach doesn't seem to do better than a shell loop consisting
of a single transfer command + sleep(1/rate)...though it is cheaper for
the client.

I think this needs a clear exposition of its use cases & commands that
attain each desired result.  And the cases that it doesn't address.


Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.

-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-users
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Received on 2022-04-04