Tests and randomness

Hi!

The other day I landed code for the test suite that allows us to run torture
tests and randomly skip a set so that more tests can go through in less time:
https://daniel.haxx.se/blog/2019/12/16/how-randomly-skipping-tests-made-them-better/

It turned out really good and I found a bunch of bugs immediately and now we
run MANY more torture tests in the CI than before.

Going forward, I would like to switch on random test order in the CI builds
(so that we make sure tests don't implicitly depend on order plus we tend to
have the "safest" tests with low numberss so randomizing the order will make
it more likely to hit "problematic" tests earlier in a run).

Two features using random, but we all know random is tricky in tests since it
makes reproducing things harder. I want to address that trickiness like this:

runtests.pl will set a random seed based on $year and $month, so the same
random order will be maintained during a whole month until it changes. This
makes repeated runs, rebases, new commits and whatever, not change the test
sequence - unless it is done over a month shift. (Work on this is in #4734)

runtests.pl will show the random seed used (in the regular test header output)
and offer an option (--seed) to set the random seed. With this option we can
set the seed to a specific one to reproduce a particular run.

The monthly seed thing is just the best way I've come up with that makes the
random order semi-stable during development and testing of something but still
changes it every now and then. I'm interested to learn if someone can come up
with a better idea that perhaps can make it change slightly more frequently
than each month...

One idea I have and I think I'll add to this, is: add a hash of the "system"
string to the seed number, so that each different system will have its own
fixed seed for the month.

Thoughts?

#4734 = https://github.com/curl/curl/pull/4734

-- 
  / daniel.haxx.se | Get the best commercial curl support there is - from me
                   | Private help, bug fixes, support, ports, new features
                   | https://www.wolfssl.com/contact/
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html