A CI job inventory

From: Daniel Stenberg via curl-library <curl-library_at_lists.haxx.se>
Date: Mon, 7 Feb 2022 23:10:39 +0100 (CET)

Hello team!

# Background

At the moment we have a little over 100 CI jobs that run for each PR push and
for each merge to the master repository on github.

The dashboard [1] shows some graphs for the CI development and changes over
time, but I really feel that I am starting to lose track of all the builds and
exactly what they verify or not.

In order to get better overview and control of the jobs we run, I'm proposing
that we create and maintain a single file that lists all the jobs we run. This
"database" of jobs could then be used to run checks against and maybe generate
some tables or charts and what not to help us make sure our CI jobs really
covers as many build combinations as possible and perhaps it can help us
reduce duplications or too-similar builds.

# The plan

Create docs/CI.md to that documents the file format for the inventory

Create tests/ci-jobs.txt which is the actual list with all the jobs.

I'll create a first PR soon when I have a few jobs added, then probably
add/convert more jobs in subsequent PRs in a batch manner.

# File format

The file format is a simple "key: value" design where "id" is the first one
for a job and then we have a long range of keys to use. See below.

- id (short unique identifier among the CI jobs). The CI job should refer back
   to this id.
- desc (optional human redable description)
- service (running it: zuul, gha, azure, cirrus, circle, appveyor)
- time (number of minutes a typical run goes)
- source (file for the CI job in the git repo)
- compiler (gcc, clang etc)
- special compiler options (sanitizers etc)
- OS (linux, windows, freebsd, macos)
- CPU (x86_64, x86, arm)
- build system (autotools, cmake, winbuild, vstudio)
- debug enabled (true/false)
- runs tests (true/false)
- valgrind (true/false)
- amissl (true/false)
- bearssl (true/false)
- gnutls (true/false)
- mbedtls (true/false)
- nss (true/false)
- openssl (true/false)
- BoringSSL (true/false)
- libressl (true/false)
- rustls (true/false)
- schannel (true/false)
- secure-transport (true/false)
- wolfssl (true/false)
- libssh2 (true/false)
- libssh (true/false)
- wolfssh (true/false)
- zlib (true/false)
- zstd (true/false)
- brotli (true/false)
- libidn2 (true/false)
- winidn (true/false)
- c-ares (true/false)

# Basic verification

I hope I can also write up some basic checks that the jobs used in this list
is also actually defined in the yaml files and vice versa, to help us make
sure that the files are all in sync even when going forward.

# Feedback

Thoughts? Anything major I've forgotten, did wrong or should add to this while
we're at it? Does the file format work or should I do it differently?

[1] = https://curl.se/dashboard.html

-- 
  / daniel.haxx.se
  | Commercial curl support up to 24x7 is available!
  | Private help, bug fixes, support, ports, new features
  | https://curl.se/support.html
-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Received on 2022-02-07