Skip to content

fix(ci): migrate 3 monitoring crons to self-hosted runners (DAK-7665) - #257

Merged
ferhimedamine merged 2 commits into
mainfrom
fix/dak7665-selfhosted-monitoring
Aug 2, 2026
Merged

fix(ci): migrate 3 monitoring crons to self-hosted runners (DAK-7665)#257
ferhimedamine merged 2 commits into
mainfrom
fix/dak7665-selfhosted-monitoring

Conversation

@ferhimedamine

Copy link
Copy Markdown
Contributor

Summary

Migrates the 3 GitHub-hosted (ubuntu-latest) monitoring cron workflows to self-hosted Hetzner runners so infra monitoring survives GitHub Actions billing outages (DAK-7664 incident, 2026-08-01 ~17:31Z).

Workflows changed:

  • Hetzner Server Health Check[self-hosted, linux, arm64] (both jobs)
    • Monitors prod server 178.104.45.161 via HTTP + Hetzner API. No self-reference risk.
  • Playground Uptime Check (DAK-7020)[self-hosted, linux, arm64]
    • HTTP probe to playground.dakera.ai. No self-reference risk.
  • Wedged Runner Auto-Heal (DAK-7528)[self-hosted, linux, x64]
    • Monitors ARM + x64 runners. Runs on x64 so it can SSH-heal ARM when ARM is wedged.
    • Partial self-reference trade-off documented in comment: if x64 itself is wedged, this job cannot run, but ARM is still covered. Partial coverage beats zero coverage during billing lock.

Verification: gh, python3, jq, curl, ssh all confirmed installed on both self-hosted runners before migration.

Test plan

  • workflow_dispatch each workflow after merge, confirm they run on self-hosted runners (not ubuntu-latest)
  • Confirm Hetzner Health Check runs on ARM runner (hetzner-arm-deploy or equivalent)
  • Confirm Playground Uptime Check runs on ARM runner
  • Confirm Wedged Runner Auto-Heal runs on x64 runner
  • Verify Telegram alert paths still functional (no tool dependency changes)

🤖 Generated with Claude Code

Move Hetzner Health Check, Wedged Runner Auto-Heal, and Playground
Uptime Check off ubuntu-latest (GitHub-hosted) onto self-hosted
Hetzner runners so monitoring survives GitHub Actions billing outages
(DAK-7664).

- hetzner-health-check.yml: both jobs → [self-hosted, linux, arm64]
  (monitors prod server 178.104.45.161, no self-reference risk)
- playground-uptime-check.yml → [self-hosted, linux, arm64]
  (HTTP probe only, no self-reference risk)
- runner-wedged-watchdog.yml → [self-hosted, linux, x64]
  (runs on x64 to monitor ARM cross-coverage; x64-watches-ARM is the
  common failure direction; updated comment documents partial trade-off)

Partial self-reference accepted per DAK-7665: coverage on self-hosted
beats zero coverage during a GitHub billing lock.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… (DAK-7665)

The 'Validate Docker Configs' CI job was on ubuntu-latest solely because
hadolint-action needs Docker auth to pull ghcr.io/hadolint/hadolint. This
caused CI to fail entirely during the 2026-08-01 GitHub billing lock (DAK-7664).

Fix: run on [self-hosted, linux, arm64] + explicit docker/login-action@v4
step to authenticate Docker on the runner to GHCR using GITHUB_TOKEN
(packages: read). Docker and docker compose are both available on the ARM
runner (Docker 29.1.3, Compose v5.1.1 verified 2026-08-01).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ferhimedamine
ferhimedamine merged commit ca1fe5a into main Aug 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant