Skip to content

Commit b54b4b1

Browse files
committed
fix(ci): sweep fixture-named VPCs, not just untagged ones
The orphan-VPC sweep selected only VPCs carrying no Name tag. Every VPC the conformance harness creates out of band is named with a fixture prefix, so the sweep skipped all of them and could only ever collect the few whose tagging had failed outright. Transit gateway fixtures leak a VPC most runs: their teardown loses the race against transit gateway attachment deletion, which holds the VPC until the attachment is gone. Nothing ever collected the remainder. By 2026-08-17 eighteen had banked up against a quota of twenty, at which point fixtures that create their own VPC started failing on VPC quota rather than on anything they had done. Cloud Map fixtures took the brunt, since a private DNS namespace needs a VPC; servicediscovery-service failed every conformance nightly for a week, and the rest of the red jobs were whichever fixture happened to ask for a VPC after the quota filled. Selection stays tag-based rather than becoming "every non-default VPC". The account can legitimately hold VPCs no test run owns, and a future parallel matrix needs a sweep that can narrow to the resources one run owns rather than one that empties the region. A VPC now qualifies when it has no Name tag at all, or when its Name matches a fixture prefix. Scoping a sweep to a single run needs more than the prefix: the harness stamps a per-resource random suffix but no run identity, so two concurrent runs cannot tell their VPCs apart. That is a harness change and is tracked separately.
1 parent 1d3b689 commit b54b4b1

1 file changed

Lines changed: 18 additions & 2 deletions

File tree

scripts/ci/clean-environment.sh

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1183,7 +1183,7 @@ done
11831183
echo "Cleaning orphaned untagged VPCs..."
11841184
DEFAULT_VPC=$(aws ec2 describe-vpcs --region "$REGION" --filters "Name=isDefault,Values=true" --query "Vpcs[0].VpcId" --output text 2>/dev/null)
11851185

1186-
# clean_orphan_vpc: delete all dependencies of a single untagged VPC, then
1186+
# clean_orphan_vpc: delete all dependencies of a single orphaned VPC, then
11871187
# delete the VPC itself. Called via xargs -P so multiple VPCs are processed
11881188
# concurrently. All variables it needs (REGION, DEFAULT_VPC) must be exported
11891189
# by the caller — see the export block below.
@@ -1438,8 +1438,24 @@ export REGION DEFAULT_VPC
14381438
# serially meant 8-10 orphans took 50+ minutes. With -P 8 that collapses
14391439
# to roughly one VPC's worth of wait time. Output from concurrent VPCs will
14401440
# interleave, but each VPC logs its own ID so the output remains readable.
1441+
#
1442+
# Selection is tag-based, and deliberately does not sweep every non-default
1443+
# VPC: the account can hold VPCs that belong to nobody's test run, and a
1444+
# future parallel matrix needs a sweep that can be narrowed to the resources
1445+
# a run owns rather than one that empties the region. A VPC qualifies when
1446+
# it carries no Name tag at all (its tagging failed, so nothing else will
1447+
# ever identify it) or when its Name tag matches a fixture prefix.
1448+
#
1449+
# The prefix arm is what stops transit-gateway fixture VPCs accumulating.
1450+
# Their Name tag is set, so the untagged arm alone skipped them on every
1451+
# run, while their own teardown routinely loses the race against transit
1452+
# gateway attachment deletion and leaves the VPC behind. Eighteen of them
1453+
# had banked up against a quota of twenty by 2026-08-17, which is why
1454+
# fixtures that create their own VPC (the Cloud Map ones especially) began
1455+
# failing on VPC quota rather than on anything they had done wrong.
14411456
aws ec2 describe-vpcs --region "$REGION" \
1442-
--query "Vpcs[?!(Tags[?Key=='Name'])].VpcId" --output text 2>/dev/null | tr '\t' '\n' | \
1457+
--query "Vpcs[?!(Tags[?Key=='Name']) || Tags[?Key=='Name' && (contains(Value, '$TEST_PREFIX') || contains(Value, '$SDK_PREFIX') || contains(Value, '$LEGACY_LB_PREFIX'))]].VpcId" \
1458+
--output text 2>/dev/null | tr '\t' '\n' | \
14431459
grep -v "^$DEFAULT_VPC$" | grep -v "^$" | \
14441460
xargs -P 8 -I {} bash -c 'clean_orphan_vpc "$@"' _ {} || true
14451461

0 commit comments

Comments
 (0)