When an ECS Service (or other dependency, like a Lambda Function) is invalidated after changing the parameters, we implement this in CloudFormation by "updating" a custom resource. In terms of the CloudFormation lifecycle, this means that we create + update SSM parameters, then restart the dependent service, and then delete SSM Parameters that have been removed. Obviously, this means that the service is restarted too early, and will detect the old parameters that should be removed :-(
We can solve this by hooking into the CloudFormation "Delete" signal, and using that to perform a second restart if necessary.
Specifically, we can:
- Ensure the resource delete is not unnecessarily invoked by having a resource ID that reflects the dependent parameters (eg. a hash of all their names)
- Have the resource delete only restart after dependent parameters have been deleted (only restart when required) by getting it to introspect it's own stack and wait for any SSM Parameter Resources that are being deleted.
h/t @v-do for finding the bug
When an ECS Service (or other dependency, like a Lambda Function) is invalidated after changing the parameters, we implement this in CloudFormation by "updating" a custom resource. In terms of the CloudFormation lifecycle, this means that we create + update SSM parameters, then restart the dependent service, and then delete SSM Parameters that have been removed. Obviously, this means that the service is restarted too early, and will detect the old parameters that should be removed :-(
We can solve this by hooking into the CloudFormation "Delete" signal, and using that to perform a second restart if necessary.
Specifically, we can:
h/t @v-do for finding the bug