feat(monitoring): weekly warning digest cronjob (ECHO-817)

[spashii]

Implements the weekly warning digest from ECHO-817. Companion to #23 (ECHO-813), which dropped severity=warning notifications.

Shape

One CronJob in the monitoring namespace:

• Schedule — 0 9 * * 1 Europe/Amsterdam (Mon 9am).
• Source — prometheus ALERTS metric over 7d, filtered to severity=warning.
• Sink — #prod-alerts via the existing webhook in monitoring-secrets.

Output shape

Sample posted message:

:bar_chart: *Warning digest — last 7d*  (3 unique alertnames)
• *HighNodeCPU* — 3 firing event(s), ~60 min total
• *IngressHighErrorRate* — 2 firing event(s), ~5 min total
• *PodRestartSpike* — 1 firing event(s), ~2 min total

Empty week prints a green-check line, not silence — easier to confirm the job actually ran.

Sort is by total firing minutes, descending. The eyeball pattern: which warnings are eating the most time, and are any of those worth promoting to critical?

How the numbers are computed

• Minutes firing — count_over_time(ALERTS{alertstate="firing", severity="warning"}[7d]) × eval-interval (15s) ÷ 60.
• Event count — changes(ALERTS_FOR_STATE{severity="warning"}[7d]) / 2, since changes() counts both 0→1 and 1→0 transitions.

Both are approximations, not exact. Good enough for "scan in 10s."

Implementation notes

• Python 3.11-slim, stdlib only (urllib). No pip layer, no third-party.
• Script is in a ConfigMap, mounted read-only. The CronJob and ConfigMap ship together in cronjob-warning-digest.yaml.
• timeZone: Europe/Amsterdam requires k8s ≥ 1.27. If the cluster is older, drop the field and switch the schedule to UTC.
• Resources: 10m CPU / 32Mi mem requested, 100m / 128Mi limit. Job typically runs <2s.

Out of scope (deliberate)

• #alerts-non-prod digest. Filed separately — once ECHO-814 lands and there are non-prod warnings to summarise, this CronJob can be cloned with a different label filter + webhook.

Verification

Ran the script offline with a mocked prometheus response — compiles clean, queries fire, slack payload formats correctly.

Confidence

Medium. Script logic is straightforward and tested offline. Three things I'd want a human eye on:

1. The prometheus changes() heuristic — for very flappy alerts (rapid on/off) the event count could over-count. Probably fine for first pass.
2. timeZone field — if the cluster is < 1.27, this needs to drop.
3. The slack webhook only renders this in #prod-alerts (the channel the existing webhook is bound to). If you want the digest somewhere else, point me at the new webhook secret.

Refs: ECHO-817

[linear]

ECHO-817