On 2/29/2016 5:39 AM, Bluflonalgul wrote:
> Hi,
>
> I'm thinking of patching curl to add the following feature I need:
>
> Interactively throttle the download speed (instead of stop Ctrl-C and
> resume with -c and --limit-rate )

TL;DR: I think an arbitrary throttle would be some work. It's not going
to be as simple as repeatedly changing
CURLOPT_MAX_{RECV,SEND}_SPEED_LARGE (data->set.max_{recv,send}_speed)
when you want the rate to change. You would not be able to address this
in the curl tool (curl) you'd have to do it in the curl library
(libcurl) which is what the tool uses.

I will try to explain why in the context of recv. Note send is pretty
much the same, unless you ever allow a rate limit and set the buffer
less than 16k, in which case there could be buffer issues with recv
making it slightly more complicated.

Internally every time there is a read from the socket libcurl updates
the average download speed [1] (it may actually do it more often but
ignore that for the moment). The average is calculated based on the
total time spent (in your case this is important). If a rate limit is
set and the average speed exceeds it then the state is switched to "too
fast" [2] which essentially pauses the transfer ('pause' may not be
semantically correct here, I can't think of a better word). When the
average limit no longer exceeds the rate limit then it switches back to
"perform" [3].

So let's say you are downloading a 100MB file in transfer A. You have a
connection that unchecked is giving you about 1MB/s down. Halfway
through transfer A (let's say you're about 50% in) you need to start and
give transfer B priority so you set a rate limit of 160k on transfer A.
Your transfer A 1MB/s download speed which is currently its average will
slowly decrease as more time passes. This is because the average is
based on total time spent (again, this is the important part) so it will
be a while before it continues to download any data at your lower limit.

data->progress.dlspeed: 1097628 , data->set.max_recv_speed: 163840
data->progress.dlspeed: 1096521 , data->set.max_recv_speed: 163840
data->progress.dlspeed: 1095392 , data->set.max_recv_speed: 163840
data->progress.dlspeed: 1093882 , data->set.max_recv_speed: 163840

[slowly dlspeed average decreases approx every second for four min]

data->progress.dlspeed: 165195 , data->set.max_recv_speed: 163840
data->progress.dlspeed: 164626 , data->set.max_recv_speed: 163840
data->progress.dlspeed: 164062 , data->set.max_recv_speed: 163840
data->progress.dlspeed: 163493 , data->set.max_recv_speed: 163840

and so minutes later it will switch back to perform mode and attempt to
continue the transfer but the transfer may be dead by then.

So clearly this design is for users who set recv/send limits once at the
beginning of a transfer and not arbitrarily in a progress function or
some such.

If you try to reproduce this on Windows small files may mask the problem
I described because Windows has some abstract buffer scaling that if you
have a smaller download like 10MB and you wait until 5MB is downloaded
and then you want to decrease a rate, the pausing in such a case may not
cause an error because internally Windows has already received it all in
some OS buffer.

So one thing for curl devs to think about is whether there is ever a
situation other than what you describe where it would be useful to add
some piecemeal recvs even when in "too fast" mode so the transfer
doesn't stop entirely. I don't know if this is worth pursuing.

If I had an assignment like yours I'd probably propose a way in libcurl
to reset the average download speed or do a sliding window. Then, if
that was accepted (and that's a big if), in private code I would use
libcurl and not the curl tool and in my progress function set
CURLOPT_RESET_AVERAGES or something and
CURLOPT_MAX_{RECV,SEND}_SPEED_LARGE any time I need to change the recv
limit.

[1]: https://github.com/curl/curl/blob/curl-7_47_1/lib/progress.c#L292-L295
[2]: https://github.com/curl/curl/blob/curl-7_47_1/lib/multi.c#L1527-L1541
[3]: https://github.com/curl/curl/blob/curl-7_47_1/lib/multi.c#L1499-L1503

-------------------------------------------------------------------
List admin: https://cool.haxx.se/list/listinfo/curl-users
FAQ: https://curl.haxx.se/docs/faq.html
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2016-03-01