Skip to content

fix(aws): converge release infrastructure exactly - #945

Open
samnan-kradle wants to merge 1 commit into
alchemy-run:mainfrom
samnan-kradle:fix/aws-release-convergence
Open

fix(aws): converge release infrastructure exactly#945
samnan-kradle wants to merge 1 commit into
alchemy-run:mainfrom
samnan-kradle:fix/aws-release-convergence

Conversation

@samnan-kradle

Copy link
Copy Markdown

Summary

  • make ECS service reconciliation wait for the exact task definition, completed rollout, settled desired/running/pending counts, old-task drain, and healthy ALB targets
  • make deletion wait for task and target drain
  • reconcile ECR tag mutability and scan-on-push for existing/adopted repositories
  • harden AWS.state() buckets with versioning, encryption, bucket-owner ownership, and full public-access blocking
  • keep every poll bounded, including a 30-minute ECS ceiling

Validation

  • bun tsc -b
  • bun run format:check
  • bun run lint:providers
  • live AWS ECR reconciliation: 3/3 passed
  • live AWS state-store convergence: 4/4 passed
  • live AWS ECS service reconciliation/recovery: 4/4 passed across bounded sequential runs
  • post-run inventory: zero alchemy-test ECR repositories, ECS clusters, or security-test S3 buckets

The live suites use an explicit AWS_ACCOUNT_ID resolved from STS because the current default AWS provider layer can deadlock while deriving it from the surrounding environment.

Comment on lines +135 to +137
// Service.deploy now blocks until the exact task definition is the sole
// completed deployment, ECS counts have settled, and every registered
// target is healthy. Prove that contract immediately out-of-band.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test change does a good job at indicating the problem solved by your change. We typically do this intentionally and prefer to resolve stabilization as far downstream as we can (for speed). But some times it is necessary to ensure a resource is stable before moving on. What did you run into specifically that triggered this change?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey it's me the same person, I was just using my corpo acc.

Anyway, I was trying to get both health and live connectivity stats for all the services everytime I deploy to make sure "everything is working together" as the goal of IaC systems - I tried to connect from my ECS server with third party services like Trigger, Neon, PostHog, etc. I'm currently using a patch of this on my local server.

I'm new to both Effect and Alchemy, I love the concept of both Effect & Alchemy, I'm trying to migrate from SST with a hell lotta custom CI/CD scripts, so that I can spend my time on some actual features not just infra, and if I'm doing something wrong please don't mind me directing to the better way of doing this.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The concrete failure was a release verifier reporting success against the previous ECS task: the replacement task started one second after verification had already finished. That allowed the workflow to attest the wrong image/task revision even though the service eventually converged.

For the ALB-backed API, the downstream capability probes must start only after ECS is serving the exact requested task definition, all previous tasks are drained, desired == running, pending == 0, and registered targets are healthy. The delete wait addresses the symmetric teardown race. This PR keeps the wait in the ECS resource boundary so every caller receives a truthful converged result; application-level probes still remain downstream and separate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants