Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: A CI job inventory
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Timothe Litt <litt_at_acm.org>
Date: Wed, 9 Feb 2022 08:46:33 -0500
On 08-Feb-22 02:39, Daniel Stenberg wrote:
> On Mon, 7 Feb 2022, Timothe Litt via curl-library wrote:
>
>> Perhaps rather than add metadata, have a utility that each CI job
>> runs to update a database at setup and/or exit.
>
> That would be super cool.
>
> It makes me curious if there isn't something like that existing
> already that maybe we don't have to invent ourselves ?
>
There probably is, but not my area. May be more work to find something
than to create what you need. The overhead of making something generic
would seem to be disincentive - there isn't that much involved. The SQT
groups I've worked with all had custom tools.
Of course, you have 1/2 of what's required - you can use a known-good
cURL in the CI jobs to send updates.
You seem to have Perl people in your group, so here's a back of the
envelope sketch:
A DB server doesn't have to be high tech; would be pretty simple to
write a Perl script that acts as a stand-alone network server;
DBI/DBD::SQLite is a small database module that gives you SQL. Or make
it a CGI under your webserver.
The DB interface takes a little study, but is fairly intuitive.
The basic standalone network server logic is a couple of dozen lines of
code; you can use SSL client auth to prevent spamming.
Either way, see github secrets
<https://docs.github.com/en/actions/security-guides/encrypted-secrets>
for keeping the private key (or password) avail.
Perl makes it easy to map client (job) updates to DB transactions - you
can send whatever format is convenient. cURL can POST a form, or send
JSON, or just a random text string to a path.
See https://metacpan.org/pod/HTTP::Daemon::SSL for a stand-alone
ssl-enabled daemon framework.
Things to consider for a schema:
* A table of job, description, prerequsites, what it covers
* A table of run events (job, timestamp, result, stats, clientid)
* A table of config items - schema version, retention limits, any
authentication
If you use SQLite, sqlite3 comes with, so you can script maintenance in
BASH rather than Perl - e.g. delete from events where julianday('now') >
( julianday(timestamp) +365 ) and result = 'fail';
Functions (paths) that come to mind:
* Register (a job)
* Start Job(name) => runid
* End(runid, result, stats, tags,...)
(You can end up with missing "end" events if the CI machine crashes or
fails aren't trapped.)
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
Received on 2022-02-09
Date: Wed, 9 Feb 2022 08:46:33 -0500
On 08-Feb-22 02:39, Daniel Stenberg wrote:
> On Mon, 7 Feb 2022, Timothe Litt via curl-library wrote:
>
>> Perhaps rather than add metadata, have a utility that each CI job
>> runs to update a database at setup and/or exit.
>
> That would be super cool.
>
> It makes me curious if there isn't something like that existing
> already that maybe we don't have to invent ourselves ?
>
There probably is, but not my area. May be more work to find something
than to create what you need. The overhead of making something generic
would seem to be disincentive - there isn't that much involved. The SQT
groups I've worked with all had custom tools.
Of course, you have 1/2 of what's required - you can use a known-good
cURL in the CI jobs to send updates.
You seem to have Perl people in your group, so here's a back of the
envelope sketch:
A DB server doesn't have to be high tech; would be pretty simple to
write a Perl script that acts as a stand-alone network server;
DBI/DBD::SQLite is a small database module that gives you SQL. Or make
it a CGI under your webserver.
The DB interface takes a little study, but is fairly intuitive.
The basic standalone network server logic is a couple of dozen lines of
code; you can use SSL client auth to prevent spamming.
Either way, see github secrets
<https://docs.github.com/en/actions/security-guides/encrypted-secrets>
for keeping the private key (or password) avail.
Perl makes it easy to map client (job) updates to DB transactions - you
can send whatever format is convenient. cURL can POST a form, or send
JSON, or just a random text string to a path.
See https://metacpan.org/pod/HTTP::Daemon::SSL for a stand-alone
ssl-enabled daemon framework.
Things to consider for a schema:
* A table of job, description, prerequsites, what it covers
* A table of run events (job, timestamp, result, stats, clientid)
* A table of config items - schema version, retention limits, any
authentication
If you use SQLite, sqlite3 comes with, so you can script maintenance in
BASH rather than Perl - e.g. delete from events where julianday('now') >
( julianday(timestamp) +365 ) and result = 'fail';
Functions (paths) that come to mind:
* Register (a job)
* Start Job(name) => runid
* End(runid, result, stats, tags,...)
(You can end up with missing "end" events if the CI machine crashes or
fails aren't trapped.)
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
- application/pgp-signature attachment: OpenPGP digital signature