Parallelism is coming
Date: Wed, 17 Apr 2019 09:41:02 +0200 (CEST)
This is just a heads up about something I'm working on.
I've started working on making the curl tool support parallel transfers. You
can check out the early work in the bagder/parallel-transfers development
branch in git. And when I say early, it means it doesn't pass the tests yet.
As a first step I've rearchitected the code to first setup all transfers and
then perform them. This will allow us to *either* do them serially like today,
or in parallel. Or a combination.
By doing this code change I get myself some challenges to maintain the
existing functionality so now it'll take me a while to make sure all tests run
Assuming I can manage to do that, the next still will be to actually introduce
parallelism as an option.
- Parallel transfers will then of course use the multi interface from the
tool instead of the easy, like today.
- I'm tempted to spend one of our last single-letter command line options for
this! (-Z probably)
- I'm sure doing the progress meter will be fun when doing N parallel
transfers. I think I'll mimic the current one but add upp all concurrent
transfers. Possibly add a new column for number of transfers or something.
- I'm interested in ideas and suggestions on what further options we need and
want to control concurrency and parallelism. Consider this command line:
"curl -Z https://example.com/image[1-1000].jpg -O" - presumably a default
setup shouldn't actually try to fire up all thouse one THOUSAND transfers
at once. For everyone's sake. But how many should it start at once?
- I *think* we can get away with the simple option of either serial or
parallel transfers and only do the URLs in parallel up to the first --next
since then a command line can actually mix, and it can do N files in
parallel first and then do another M files in parallel, if you want to have
some control of order but still have some transfers simultaneous.
What else do we need to consider?
-- / daniel.haxx.se ----------------------------------------------------------- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users Etiquette: https://curl.haxx.se/mail/etiquette.htmlReceived on 2019-04-17