Skip to content

Retry ALB CreateRule on PriorityInUse race condition#11

Open
Copilot wants to merge 3 commits intomainfrom
copilot/fix-lambda-priority-issue
Open

Retry ALB CreateRule on PriorityInUse race condition#11
Copilot wants to merge 3 commits intomainfrom
copilot/fix-lambda-priority-issue

Conversation

Copy link

Copilot AI commented Mar 16, 2026

Concurrent Lambda invocations for different ACME domains each call GetNextAvailablePriority independently, causing both to claim the same priority. The second CreateRule call fails with PriorityInUse.

Changes

  • aws/alb/updatealb.go: Added createRuleWithRetry — on PriorityInUse, waits 1s, re-fetches the next available priority (which now correctly skips the priority claimed by the competing invocation), and retries up to 10 times
  • Refactored AddListenerStaticRule and AddListenerTriggerTargetGroupRule to use createRuleWithRetry
func createRuleWithRetry(ctx context.Context, svc *elbv2.Client, listenerArn string, input *elbv2.CreateRuleInput) error {
    for i := 0; ; i++ {
        if _, err := svc.CreateRule(ctx, input); err != nil {
            var apiErr smithy.APIError
            if errors.As(err, &apiErr) && apiErr.ErrorCode() == "PriorityInUse" {
                if i >= maxPriorityRetries {
                    return fmt.Errorf("failed to create rule after %d retries: %w", maxPriorityRetries, err)
                }
                log.Printf("Priority %d is in use, retrying (%d/%d)...", *input.Priority, i+1, maxPriorityRetries)
                time.Sleep(time.Second)
                priority, err := GetNextAvailablePriority(ctx, listenerArn)
                // ...
                input.Priority = ptr.Int32(priority)
                continue
            }
            return err
        }
        return nil
    }
}
Original prompt

This section details on the original issue you should resolve

<issue_title>Race: multiple lambda invocations try to create rule with priority 1</issue_title>
<issue_description>Two ACME domains result in two separate lambda invocations (until we do #5), each creating a rule starting from priority 1, causing one lambda to fail. We could handle the priority failure and retry.

2026-03-16T21:13:25.506Z cd ** error: sdk-v2/provider2.go:572: sdk.helper_schema: invoking Lambda Function (acme-lambda-0d3a9fa): {"errorMessage":"failed to create listener static rule: operation error Elastic Load Balancing v2: CreateRule, https response error StatusCode: 400, RequestID: 24b62462-04f5-4e6b-ba9c-f954daf18fdd, PriorityInUse: Priority '1' is currently in use","errorType":"wrapError"}: provider=aws@6.83.2

Workaround: rerunning the deployment should fix it.
</issue_description>

<agent_instructions>Handle the PriorityInUse error message and retry creation of ListenerRule</agent_instructions>

Comments on the Issue (you are @copilot in this section)


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 2 commits March 17, 2026 00:02
Co-authored-by: lionello <591860+lionello@users.noreply.github.com>
Co-authored-by: lionello <591860+lionello@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix race condition for lambda invocations creating rules Retry ALB CreateRule on PriorityInUse race condition Mar 17, 2026
Copilot AI requested a review from lionello March 17, 2026 00:08
@lionello lionello marked this pull request as ready for review March 17, 2026 01:35
@lionello lionello requested a review from edwardrf March 17, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Race: multiple lambda invocations try to create rule with priority 1

2 participants