Re: Tests and randomness

this is cool - it occurred to me that we also could run reverse order
of tests (but random is fine as well ... I would suggest making order
options configurable with default being random.

the next evolutionary step after unlinking order deps might be
parallelising test runs ... related but different set of challenges
there.

Jim

On Thu, 19 Dec 2019 at 12:11, Daniel Stenberg via curl-library
<curl-library_at_cool.haxx.se> wrote:
>
> Hi!
>
> The other day I landed code for the test suite that allows us to run torture
> tests and randomly skip a set so that more tests can go through in less time:
> https://daniel.haxx.se/blog/2019/12/16/how-randomly-skipping-tests-made-them-better/
>
> It turned out really good and I found a bunch of bugs immediately and now we
> run MANY more torture tests in the CI than before.
>
> Going forward, I would like to switch on random test order in the CI builds
> (so that we make sure tests don't implicitly depend on order plus we tend to
> have the "safest" tests with low numberss so randomizing the order will make
> it more likely to hit "problematic" tests earlier in a run).
>
> Two features using random, but we all know random is tricky in tests since it
> makes reproducing things harder. I want to address that trickiness like this:
>
> runtests.pl will set a random seed based on $year and $month, so the same
> random order will be maintained during a whole month until it changes. This
> makes repeated runs, rebases, new commits and whatever, not change the test
> sequence - unless it is done over a month shift. (Work on this is in #4734)
>
> runtests.pl will show the random seed used (in the regular test header output)
> and offer an option (--seed) to set the random seed. With this option we can
> set the seed to a specific one to reproduce a particular run.
>
> The monthly seed thing is just the best way I've come up with that makes the
> random order semi-stable during development and testing of something but still
> changes it every now and then. I'm interested to learn if someone can come up
> with a better idea that perhaps can make it change slightly more frequently
> than each month...
>
> One idea I have and I think I'll add to this, is: add a hash of the "system"
> string to the seed number, so that each different system will have its own
> fixed seed for the month.
>
> Thoughts?
>
> #4734 = https://github.com/curl/curl/pull/4734
>
> --
>
> / daniel.haxx.se | Get the best commercial curl support there is - from me
> | Private help, bug fixes, support, ports, new features
> | https://www.wolfssl.com/contact/
> -------------------------------------------------------------------
> Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
> Etiquette: https://curl.haxx.se/mail/etiquette.html
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2019-12-19