Re: A CI job inventory

From: Timothe Litt <litt_at_acm.org>
Date: Mon, 7 Feb 2022 17:31:19 -0500

Should probably support multi-line values - e.g. description.

I see backends, but where are the protocols?  E.g. does a job test
http(s)?  FTP?  etc.

Consider how to measure effectiveness - 100 seems like a lot. coverage? 
How many bugs found?  Recently?

May be worth running most effective first, or tagging PRs with area
affected so CI can test that first.  Or running less effective jobs less
frequently...


Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.

On 07-Feb-22 17:10, Daniel Stenberg via curl-library wrote:
> Hello team!
>
> # Background
>
> At the moment we have a little over 100 CI jobs that run for each PR
> push and for each merge to the master repository on github.
>
> The dashboard [1] shows some graphs for the CI development and changes
> over time, but I really feel that I am starting to lose track of all
> the builds and exactly what they verify or not.
>
> In order to get better overview and control of the jobs we run, I'm
> proposing that we create and maintain a single file that lists all the
> jobs we run. This "database" of jobs could then be used to run checks
> against and maybe generate some tables or charts and what not to help
> us make sure our CI jobs really covers as many build combinations as
> possible and perhaps it can help us reduce duplications or too-similar
> builds.
>
> # The plan
>
> Create docs/CI.md to that documents the file format for the inventory
>
> Create tests/ci-jobs.txt which is the actual list with all the jobs.
>
> I'll create a first PR soon when I have a few jobs added, then
> probably add/convert more jobs in subsequent PRs in a batch manner.
>
> # File format
>
> The file format is a simple "key: value" design where "id" is the
> first one for a job and then we have a long range of keys to use. See
> below.
>
> - id (short unique identifier among the CI jobs). The CI job should
> refer back
>   to this id.
> - desc (optional human redable description)
> - service (running it: zuul, gha, azure, cirrus, circle, appveyor)
> - time (number of minutes a typical run goes)
> - source (file for the CI job in the git repo)
> - compiler (gcc, clang etc)
> - special compiler options (sanitizers etc)
> - OS (linux, windows, freebsd, macos)
> - CPU (x86_64, x86, arm)
> - build system (autotools, cmake, winbuild, vstudio)
> - debug enabled (true/false)
> - runs tests (true/false)
> - valgrind (true/false)
> - amissl (true/false)
> - bearssl (true/false)
> - gnutls (true/false)
> - mbedtls (true/false)
> - nss (true/false)
> - openssl (true/false)
> - BoringSSL (true/false)
> - libressl (true/false)
> - rustls (true/false)
> - schannel (true/false)
> - secure-transport (true/false)
> - wolfssl (true/false)
> - libssh2 (true/false)
> - libssh (true/false)
> - wolfssh (true/false)
> - zlib (true/false)
> - zstd (true/false)
> - brotli (true/false)
> - libidn2 (true/false)
> - winidn (true/false)
> - c-ares (true/false)
>
> # Basic verification
>
> I hope I can also write up some basic checks that the jobs used in
> this list is also actually defined in the yaml files and vice versa,
> to help us make sure that the files are all in sync even when going
> forward.
>
> # Feedback
>
> Thoughts? Anything major I've forgotten, did wrong or should add to
> this while we're at it? Does the file format work or should I do it
> differently?
>
> [1] = https://curl.se/dashboard.html
>
>

-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Received on 2022-02-07