Feat: metrics summary

[MarceloRobert]

Adds a metrics summary notification (just the template for now, WIP)

Example output:

KernelCI Metrics Summary
========================
Period: 2025-11-26 17:11 UTC to 2025-12-03 17:11 UTC


ACTIVITY
--------
    53 issues
   530 incidents
    12 checkouts
    40 builds
  6000 tests


REGRESSIONS
-----------
Incidents are occurrences of an issue.
New regressions are the first incident of an issue.
Only build issues considered.

Origin      Total Incidents    New Regressions
──────────────────────────────────────────────
origin1     30                 5
origin2     500                253


LAB ACTIVITY
------------
There were 2 labs registered. Incidents reported per lab:

Lab (Origin)        Builds    Boots     Tests
─────────────────────────────────────────────
lab1 (origin1)      5         2         0
lab2 (origin2)      0         4         10
─────────────────────────────────────────────
Total               5         6         10



--
This is an experimental report format. Please send feedback in!
Talk to us at [email protected]

Made with love by the KernelCI team - https://kernelci.org

[barbieri]

I'd use sorted(lab_map.items() so it's stable (sorted by key)

[MarceloRobert]

Good point. I couldn't use sorted() itself because jinja doesn't support it, but I found that there is the | sort filter that is basically the same.
When I checked that filter I also found that I could replace total_labs with just lab_maps | length (jinja doesn't support len() either)

[padovan]

Issues and incidents are the complicated part I believe. For issues we could start only with the build ones and deploy the report with that info. And then be clear on the language that incidents refers to issues.

Frequency could be weekly at first to iterate faster? And for a limited audience I guess.

[bhcopeland]

Looks good. A few things that confuse me, the use of 'incidents' and 'regressions'.

• Regression = a new issue (first time this problem was seen)
• Incident = any occurrence of an issue (including recurring ones)

So "5 new regressions" + "25 recurring incidents" = "30 total incidents"

The total then is a bit confusing, I see "X new incidents" listed in the breakdown and the total.

I had a little play with the report and changed the formatting:

  KernelCI Metrics Summary
  ========================
  Period: 2025-11-25 18:26 to 2025-12-02 18:26


  ACTIVITY
  --------
      12 checkouts
      40 builds
    6000 tests (boot + non-boot)


  REGRESSIONS
  -----------
  5 new regressions detected (issues with their first incident this period)

  30 incidents total, broken down by type:
       2 build failures
       5 boot failures
      23 test failures

  53 issues tracked (includes recurring issues from previous periods)


  LAB ACTIVITY
  ------------
  Incidents reported per lab:

      Lab                   Build    Boot    Test
      ─────────────────────────────────────────────
      lab1 (origin1)            5       2       0
      lab2 (origin2)            0       0      10
      ─────────────────────────────────────────────
      Total                     5       2      10

I know its just draft, but I see there is setup_jinja_template() already in the codebase, maybe this could be reused? Or, if not maybe the generate_metrics_report() can be moved into notifications.py

[MarceloRobert]

I know its just draft, but I see there is setup_jinja_template() already in the codebase, maybe this could be reused? Or, if not maybe the generate_metrics_report() can be moved into notifications.py
@bhcopeland

I didn't want to add this code to notifications.py since notifications.py is already a pretty big file with functions for multiple actions, but then I couldn't use the setup_jinja_template() in this new file because it would create a circular dependency and the subfolder that contains the template is different. But I'm already checking on a small refactor just to move that helper function out of notifications.py so that I can reuse it

[MarceloRobert]

Issues and incidents are the complicated part I believe. For issues we could start only with the build ones and deploy the report with that info. And then be clear on the language that incidents refers to issues.
@padovan

Looks good. A few things that confuse me, the use of 'incidents' and 'regressions'.

• Regression = a new issue (first time this problem was seen)
• Incident = any occurrence of an issue (including recurring ones)
  So "5 new regressions" + "25 recurring incidents" = "30 total incidents"

The total then is a bit confusing, I see "X new incidents" listed in the breakdown and the total.
@bhcopeland

What about something like this:

- 5 new regressions (issues that had their first incident in the given interval).
-
-
- 30 total incidents in this interval, being:
-     2 build incidents
-     5 boot incidents
-     23 test incidents
+ 20 build incidents in total, 3 of which are new regressions (the first incident of an issue)

(made up numbers unrelated to the prior example)

I don't want to just say "failures" because I'm not gathering builds that failed without any related issue. And is it useful to separate these incidents/regressions by origin?

Btw thanks @bhcopeland for the formatting suggestion. I'll make some modifications and push changes today