fix(runner-watchdog): use RUNNER_SSH_KEY for runner host SSH (DAK-7637) - #256
Merged
Conversation
DEPLOY_SSH_KEY is the prod-server key and is not authorized on the runner hosts (168.119.60.30 ARM, 178.104.227.173 x64). The ops@dakera.ai ED25519 key is what's in authorized_keys on both runner hosts. - Switch Setup SSH step from DEPLOY_SSH_KEY → RUNNER_SSH_KEY - RUNNER_SSH_KEY secret added to dakera-deploy repo today - Remove "Manual intervention required" from failure alert — Platform self-heals via issue assignment, never escalates to founder Co-Authored-By: Platform Agent <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The wedged-runner watchdog was using
DEPLOY_SSH_KEYto SSH into the runner hosts. This key is authorized on the prod server (178.104.45.161) but NOT on the runner hosts:RUNNER_SSH_KEY(ops@dakera.ai) + deploy key is NOT in authorized_keysRUNNER_SSH_KEY(ops@dakera.ai)Result: both SSH connections returned
Permission denied (publickey), the watchdog couldn't restart the runners, and sent a "Manual intervention required" Telegram to the founder.Fix
RUNNER_SSH_KEY— added to dakera-deploy repo (ops@dakera.ai ED25519, authorized on both runner hosts). Already set out-of-band.RUNNER_SSH_KEYinstead ofDEPLOY_SSH_KEYfor runner host SSH.Verification
ssh -i ~/.ssh/id_ed25519 root@168.119.60.30✅RUNNER_SSH_KEYsecret set in dakera-deploy ✅Root cause trail
DAK-7637 alert → watchdog run 30620415976 →
Permission denied (publickey)on both hosts → wrong key secret used.Co-Authored-By: Platform Agent noreply@anthropic.com