Re: curl-users Digest, Vol 164, Issue 12
Date: Sat, 27 Apr 2019 15:35:59 -0400
On 27-Apr-19 06:00, curl-users-request_at_cool.haxx.se wrote:
> Message: 3
> Date: Fri, 26 Apr 2019 23:27:09 +0200 (CEST)
> From: Daniel Stenberg <daniel_at_haxx.se>
> To: the curl tool <curl-users_at_cool.haxx.se>
> Subject: [STATUS UPDATE] Parallelism
> Message-ID: <alpine.DEB.2.20.1904262301350.5067_at_tvnag.unkk.fr>
> Content-Type: text/plain; format=flowed; charset=US-ASCII
> Hi again,
> Since I got even further today, I just wanted to share with you all how this
> new functionality works right now and possibly trick one or two of you to try
> it out or provide some thoughts and ideas!
> - The status meter alone feels like something I could write a lot about. The
> first point perhaps being that we can't use the "normal" progress meter/bars
> when doing parallel transfers. I've implemented a completely new one that
> is designed to handle any amount of transfers. It's a little tricky. It
> currently shows:
> Here's an example progress meter snapshot. It's me asking for 101 transfers,
> consisting of in total 52.7GB of data. Asking for '--parallel-max 30' it
> means that there will never be more than 30 "live" transfers. In this case
> here, 48 transfers are already completed:
> DL% UL% Dled Uled Xfers Live Qd Total Current Left Speed
> 72 -- 37.9G 0 101 30 23 0:00:55 0:00:34 0:00:22 2752M
> Good enough? What's missing here that we need?
Great start. You asked for comments, so...
I think I want to know about the outliers/problems, not so much the
How about at least #failed transfers? That would fit on a 1-line status.
# stalled (no data sent/received in - pick a threshold, say 5 secs?)
could be helpful
The issue with "speed" is that while it tells you about curl's
performance, what a
user wants to know is what needs attention - which transfers are NOT making
progress, or are slow relative to the pack e.g. (where picking a
different mirror might help).
> o percent download (if known, which means *all* transfers need to have a
> known size)
The more files are involved, the greater the chance that at least one
doesn't have a known
size. Why not compute the statistic for all the transfers that do?
Then add an asterisk
* Excluding 17 transfers whose size is unknown (or not yet known)
You have to be doing essentially the same thing for queued transfers....
> Work pending:
> - Ponder if there's any way we can "report" transfer status/success for
> individual transfers here that makes sense to users. Doing 101 transfers
> is all fun and games, but if one of the transfers failed, surely a user
> would like a way to figure this out?
How about a curses interface? That would give you a status window where
you can put
detailed information on each transfer - perhaps just one line each.
curses windows can
randomly update & can scroll, so you can have an "unlimited" list.
You might consider several windows: queued, active, complete & failed.
That lets a user
focus on any failures that need typos/permissions fixed; complete that
let him proceed;
active to keep him entertained; and queued to keep him frustrated :-)
For you, it means
that you can afford the space to provide details appropriate to the
state - rather than try
to cram all possible state into one line format.
Pretty much every environment these days has (or can be connected to) a
command window. It's not like when a VT100 was a several thousand
I'm pretty sure everyone has run 'top', less or vi in a command window -
you don't need
to go wild with video.
I think curses is a reasonable balance between a pure command line tool
and the popular overdone, platform-dependent graphics. I don't think
you want to go there.
If you are unwilling to require curses support, another approach is to
build a status page
as described, and define an key to print it. So instead of continuously
you get a detailed dump when you press a key (say space). You could
also have a set
of pages/keys: Q for queued transfer list, F for failed transfers, C for
completed transfers, and
<space> for active transfers. In that way one gets status on demand,
but you don't have to
worry about multi-line cursor addressing.
No matter what you do, consider accessibility - try running your code
with a screen reader.
The "press for status" would be very friendly for accessibility as once
displayed, the information
is static. Otherwise, I prefer curses.
> - Tests. Tricky, but I have not gotten to this yet.
> - make --retry work for parallel
> - Consider a --parallel-host-max to limit the number of connections done to
> a single host but I also think that it is better saved as a separate PR
> for once the initial support lands.
> -- / daniel.haxx.se
- application/pgp-signature attachment: OpenPGP digital signature