Skip to content

fix(runner-watchdog): use RUNNER_SSH_KEY for runner host SSH (DAK-7637) - #256

Merged
ferhimedamine merged 1 commit into
mainfrom
fix/dak7637-runner-ssh-key
Jul 31, 2026
Merged

fix(runner-watchdog): use RUNNER_SSH_KEY for runner host SSH (DAK-7637)#256
ferhimedamine merged 1 commit into
mainfrom
fix/dak7637-runner-ssh-key

Conversation

@ferhimedamine

Copy link
Copy Markdown
Contributor

Problem

The wedged-runner watchdog was using DEPLOY_SSH_KEY to SSH into the runner hosts. This key is authorized on the prod server (178.104.45.161) but NOT on the runner hosts:

  • ARM runner (168.119.60.30): only RUNNER_SSH_KEY (ops@dakera.ai) + deploy key is NOT in authorized_keys
  • x64 runner (178.104.227.173): only RUNNER_SSH_KEY (ops@dakera.ai)

Result: both SSH connections returned Permission denied (publickey), the watchdog couldn't restart the runners, and sent a "Manual intervention required" Telegram to the founder.

Fix

  1. New secret RUNNER_SSH_KEY — added to dakera-deploy repo (ops@dakera.ai ED25519, authorized on both runner hosts). Already set out-of-band.
  2. Watchdog uses RUNNER_SSH_KEY instead of DEPLOY_SSH_KEY for runner host SSH.
  3. Alert message — replaced "Manual intervention required — SSH into..." with "Platform agent will investigate and self-heal." Platform owns runner recovery; founder should never be asked to SSH.

Verification

  • Runner hosts confirmed reachable with ops key: ssh -i ~/.ssh/id_ed25519 root@168.119.60.30
  • ARM runner restarted manually (DAK-7637), reconnected to GitHub, now processing previously-queued CI job (run 30619220135) ✅
  • RUNNER_SSH_KEY secret set in dakera-deploy ✅

Root cause trail

DAK-7637 alert → watchdog run 30620415976Permission denied (publickey) on both hosts → wrong key secret used.

Co-Authored-By: Platform Agent noreply@anthropic.com

DEPLOY_SSH_KEY is the prod-server key and is not authorized on the
runner hosts (168.119.60.30 ARM, 178.104.227.173 x64). The ops@dakera.ai
ED25519 key is what's in authorized_keys on both runner hosts.

- Switch Setup SSH step from DEPLOY_SSH_KEY → RUNNER_SSH_KEY
- RUNNER_SSH_KEY secret added to dakera-deploy repo today
- Remove "Manual intervention required" from failure alert — Platform
  self-heals via issue assignment, never escalates to founder

Co-Authored-By: Platform Agent <noreply@anthropic.com>
@ferhimedamine
ferhimedamine merged commit bb8efea into main Jul 31, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant