Experimenting with parallel tests on Debian

From: Samuel Henrique via curl-library <curl-library_at_lists.haxx.se>
Date: Thu, 11 Jan 2024 10:44:42 -0300

Hello,

I have recently pushed an experimental build of curl with parallel test
execution on Debian. This was done with the hopes of helping reporting issues
and understanding if it's feasible to enable it for non-experimental builds.

We have quite a diverse set of supported architectures and different build
hosts[0].

There were a few failures that went away after retries. I have not done any
investigation other than noting the failed tests were not always the same and
at least one failure occurred on a host with a high number of CPU threads (16,
high-ish for non-server standards nowadays).

All the builds were done following the suggestion of using 7 workers per CPU
thread [1] and without valgrind.

Do note that I did not try a lower number of workers and I'm only sending this
in case someone is interested in finding possible bugs. I plan to keep testing
future releases and me or someone else from Debian might report something more
concrete in the future.

--------

hurd-i386, no retries were done:
1 thread, built with -j1 and tested with -j7.

> TESTFAIL: These test cases failed: 532 527 540 575 1501
Logs for the failure:
https://buildd.debian.org/status/fetch.php?pkg=curl&arch=hurd-i386&ver=8.5.0-2%2Bexp1&stamp=1704551911&raw=0

--------

i386, retry succeeded on a different host:
Both hosts have 6 threads, built with -j6, tested with -j42.

> TESTFAIL: These test cases failed: 237
Logs for the failure:
https://buildd.debian.org/status/fetch.php?pkg=curl&arch=i386&ver=8.5.0-2%2Bexp1&stamp=1704545546&raw=0

All logs (click the value in the "Result" column for logs):
https://buildd.debian.org/status/logs.php?pkg=curl&ver=8.5.0-2%2Bexp1&arch=i386


--------

powerpc, retry succeeded on a different host:
Failed on host with 16 threads, built with -j16, tested with -j112.
Succeeded on host with 32 threads, built with -j32, tested with -j224.

> TESTFAIL: These test cases failed: 244
Logs for the failure:
https://buildd.debian.org/status/fetch.php?pkg=curl&arch=powerpc&ver=8.5.0-2%2Bexp1&stamp=1704547069&raw=0

All logs (click the value in the "Result" column for logs):
https://buildd.debian.org/status/logs.php?pkg=curl&ver=8.5.0-2%2Bexp1&arch=powerpc

--------

x32, second retry succeeded, different host:
Both hosts have 2 threads, built with -j2, tested with -j14.

> TESTFAIL: These test cases failed: 352
Logs for the failure:
https://buildd.debian.org/status/fetch.php?pkg=curl&arch=x32&ver=8.5.0-2%2Bexp1&stamp=1704546530&raw=0

> TESTFAIL: These test cases failed: 144 348
Logs for the failure:
https://buildd.debian.org/status/fetch.php?pkg=curl&arch=x32&ver=8.5.0-2%2Bexp1&stamp=1704637758&raw=0

All logs (click the value in the "Result" column for logs):
https://buildd.debian.org/status/logs.php?pkg=curl&ver=8.5.0-2%2Bexp1&arch=x32

[0] https://buildd.debian.org/status/package.php?p=curl&suite=experimental
[1] https://daniel.haxx.se/blog/2023/06/08/parallel-curl-tests/

--
Samuel Henrique <samueloph>
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2024-01-11