Skip to content

infra: silence nag when dev3 lights are off#2985

Merged
cqnykamp merged 2 commits into
Doenet:mainfrom
cqnykamp:infra/dev3-health-check-lights
Jul 1, 2026
Merged

infra: silence nag when dev3 lights are off#2985
cqnykamp merged 2 commits into
Doenet:mainfrom
cqnykamp:infra/dev3-health-check-lights

Conversation

@cqnykamp

@cqnykamp cqnykamp commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Summary

When the dev3 service is scaled to zero by the lights-out flow, the Route53 health-check alarm predictably transitions to ALARM and — because it matches the ${env}-${service}-*-Extended-* wildcard — triggers the nag pipeline (EventBridge → Step Function → SNS critical topic → 5-minute re-page loop). This PR gates that pipeline on lights state so scheduled downtime doesn't page on-call.

What changed

  • LightsOut lambda now calls events.disable_rule (in us-east-1) on the ${env}-${service}-nag-extended-alarms EventBridge rule — the rule that forwards alarm state-change events from us-east-1 to the us-east-2 default event bus. Once disabled, no state-change events reach NagStateMachine, so no SNS notifications fire.
  • LightsOn lambda re-enables the rule.
  • Lambda IAM policy gains events:DisableRule and events:EnableRule.

Silencing at the forwarding-rule level (rather than the alarm's AlarmActions) is what actually stops the nag: EventBridge CloudWatch Alarm State Change events fire regardless of ActionsEnabled, so DisableAlarmActions alone would have left the nag intact.

Test plan

Verified on dev3:

  • Rule starts in ENABLED (aws events describe-rule --name dev3-doenet-nag-extended-alarms --region us-east-1).
  • Lights off → LightsOut lambda logs show no IAM errors → rule flips to DISABLED.
  • Health-check alarm transitions to ALARM as expected, but no new dev3-nag-state-machine executions start and no critical SNS notification is delivered.
  • Lights on → rule flips back to ENABLED, LightsOn lambda logs clean.

@cqnykamp cqnykamp changed the title infra: gate extended health check alarm on service lights state infra: silence nag when dev3 lights are off Jul 1, 2026
@cqnykamp cqnykamp merged commit 8a971e5 into Doenet:main Jul 1, 2026
12 checks passed
@cqnykamp cqnykamp deleted the infra/dev3-health-check-lights branch July 1, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant