On 27-Apr-19 06:00, curl-users-request_at_cool.haxx.se wrote:
> Message: 3
> Date: Fri, 26 Apr 2019 23:27:09 +0200 (CEST)
> From: Daniel Stenberg <daniel_at_haxx.se>
> To: the curl tool <curl-users_at_cool.haxx.se>
> Subject: [STATUS UPDATE] Parallelism
> Message-ID: <alpine.DEB.2.20.1904262301350.5067_at_tvnag.unkk.fr>
> Content-Type: text/plain; format=flowed; charset=US-ASCII
>
> Hi again,
>
> Since I got even further today, I just wanted to share with you all how this
> new functionality works right now and possibly trick one or two of you to try
> it out or provide some thoughts and ideas!
>
> [Snip]
> - The status meter alone feels like something I could write a lot about. The
> first point perhaps being that we can't use the "normal" progress meter/bars
> when doing parallel transfers. I've implemented a completely new one that
> is designed to handle any amount of transfers. It's a little tricky. It
> currently shows:
>
>
> Here's an example progress meter snapshot. It's me asking for 101 transfers,
> consisting of in total 52.7GB of data. Asking for '--parallel-max 30' it
> means that there will never be more than 30 "live" transfers. In this case
> here, 48 transfers are already completed:
>
> DL% UL% Dled Uled Xfers Live Qd Total Current Left Speed
> 72 -- 37.9G 0 101 30 23 0:00:55 0:00:34 0:00:22 2752M
>
> Good enough? What's missing here that we need?

Great start.  You asked for comments, so...

I think I want to know about the outliers/problems, not so much the
successful
transfers.

How about at least #failed transfers?  That would fit on a 1-line status.

# stalled (no data sent/received in - pick a threshold, say 5 secs?)
could be helpful

The issue with "speed" is that while it tells you about curl's
performance, what a
user wants to know is what needs attention - which transfers are NOT making
progress, or are slow relative to the pack e.g. (where picking a
different mirror might help).

> o percent download (if known, which means *all* transfers need to have a
> known size)
The more files are involved, the greater the chance that at least one
doesn't have a known
size.  Why not compute the statistic for all the transfers that do? 
Then add an asterisk

DL%

42*

* Excluding 17 transfers whose size is unknown (or not yet known)

You have to be doing essentially the same thing for queued transfers....

> Work pending:
>
> - Ponder if there's any way we can "report" transfer status/success for
> individual transfers here that makes sense to users. Doing 101 transfers
> is all fun and games, but if one of the transfers failed, surely a user
> would like a way to figure this out?

How about a curses interface?  That would give you a status window where
you can put
detailed information on each transfer - perhaps just one line each. 
curses windows can
randomly update & can scroll, so you can have an "unlimited" list. 

You might consider several windows: queued, active, complete & failed. 
That lets a user
focus on any failures that need typos/permissions fixed; complete that
let him proceed;
active to keep him entertained; and queued to keep him frustrated :-) 
For you, it means
that you can afford the space to provide details appropriate to the
state - rather than try
to cram all possible state into one line format.

Pretty much every environment these days has (or can be connected to) a
curses-capable
command window.  It's not like when a VT100 was a several thousand
dollar item...

I'm pretty sure everyone has run 'top', less or vi in a command window -
you don't need
to go wild with video.

I think curses is a reasonable balance between a pure command line tool
(curl's tradition),
and the popular overdone, platform-dependent graphics.  I don't think
you want to go there.

If you  are unwilling to require curses support, another approach is to
build a status page
as described, and define an key to print it.  So instead of continuously
updating status,
you get a detailed dump when you press a key (say space).  You could
also have a set
of pages/keys: Q for queued transfer list, F for failed transfers, C for
completed transfers, and
<space> for active transfers.  In that way one gets status on demand,
but you don't have to
worry about multi-line cursor addressing.

No matter what you do, consider accessibility - try running your code
with a screen reader.

The "press for status" would be very friendly for accessibility as once
displayed, the information
is static.  Otherwise, I prefer curses.

> - Tests. Tricky, but I have not gotten to this yet.
>
> - make --retry work for parallel
>
> - Consider a --parallel-host-max to limit the number of connections done to
> a single host but I also think that it is better saved as a separate PR
> for once the initial support lands.
>
> -- / daniel.haxx.se

-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html