test(e2e): migrate a released workspace to the local build - #1002
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1002 +/- ##
=======================================
Coverage 88.59% 88.59%
=======================================
Files 180 180
Lines 6699 6699
Branches 1622 1622
=======================================
Hits 5935 5935
Misses 364 364
Partials 400 400 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
nx-plugin-for-aws
force-pushed
the
feat/e2e-migrate-harness
branch
from
July 31, 2026 04:45
0c1894c to
aa9a245
Compare
cogwirrel
reviewed
Jul 31, 2026
nx-plugin-for-aws
force-pushed
the
feat/e2e-migrate-harness
branch
from
August 1, 2026 05:20
807feb2 to
0353eb2
Compare
Adds a `migrate` smoke test that upgrades a workspace generated by a released version of the plugin to the local build using the real `nx migrate` cycle, so the migration collection is regressed against a workspace a user actually has rather than only against synthetic trees in unit tests. Closes #705
- Run the hops under pnpm rather than npm, relaxing peer strictness with pnpm's own `strict-peer-dependencies=false` instead of `legacy-peer-deps`. - Scaffold with `internal#test-matrix` so each hop gets the start version's own full generator coverage. Releases predating that generator keep the fixed recipe; the fallback goes when the supported range no longer reaches them. - Migrate from the last 5 releases rather than only the latest. - Move `publishForMigrateSmokeTest` into `migrate-versions`, alongside the version resolution it depends on, and skip it unless the run includes the migrate test — every lane shares global-setup, and mirroring a release per start version is wasted work for the lanes that never read it. - Drop the user-customisation assertion, which hardcoded a path from the recipe's website project and kept the driver from staying generic. Assert the upgraded version against the installed package rather than the manifest range: on a pnpm workspace the manifest holds `catalog:` and the version lives in `pnpm-workspace.yaml`, so node_modules is the package-manager-agnostic record — and what the migrations actually run from.
Replace the hardcoded `FIRST_VERSION_WITH_TEST_MATRIX` gate with a runtime capability check: the hop runs `nx list @aws/nx-plugin` against the workspace it just created and looks for `internal#test-matrix`. No release boundary is encoded, and the answer comes from the collection the hop is about to generate from. A start version that does not vend it is skipped with a warning rather than falling back to a hand-maintained recipe, so the fixed recipe is gone. Verified the check by publishing two plugin builds to a local verdaccio, one with `internal#test-matrix` and one with it stripped, then running the predicate against a workspace on each: true and false respectively. Also verified against a real 1.0.0-rc.50 workspace built from the published tarball, which correctly reports false. Also assert the "before" workspace is actually on the start version before migrating. `nx migrate` only runs migrations above the installed version, so a workspace that silently resolved the local `0.0.0` build (published to verdaccio as `latest`) yields a hop that migrates nothing and still passes. This assertion fires today, which is a pre-existing hole in the harness rather than a regression: the log shows create selecting `preset=@aws/nx-plugin@1.0.0-rc.50` while the resulting catalog pins `0.0.0`, so the executed preset was the local build. Left failing deliberately rather than asserting a hop that proves nothing.
`@nx/workspace`'s new generator resolves the preset's version as `NX_E2E_PRESET_VERSION ?? <version parsed from --preset>` — the environment variable takes priority. `global-setup` sets it to the local build's version so the other smoke tests install that, which silently overrode the version pin the migrate test passes: `create` reported `preset=@aws/nx-plugin@1.0.0-rc.50` while the workspace was scaffolded and installed at `0.0.0`. Because `nx migrate` only runs migrations whose version is above the installed one, every hop was really `0.0.0 -> target`, leaving every migration out of range and asserting nothing about a real upgrade. The lane was green because it was vacuous. `createTestWorkspace` now points `NX_E2E_PRESET_VERSION` at the version it was asked for, so the variable can no longer contradict the pin. Only the migrate test passes a version, so no other lane changes behaviour. Verified end to end: the workspace now installs 1.0.0-rc.50, which the assertion added alongside this confirms — it failed before this change and passes after.
Until a release ships `internal#test-matrix`, every hop skips itself and the lane passes having asserted nothing. Log which start versions migrated and how many skipped, and say explicitly when none ran, so a green tick can't imply coverage that isn't there.
The matrix runs the license generator, whose dependency allowlist rejects some of what the matrix itself pulls in (`mariadb` is LGPL-2.1-or-later, via the mysql rdb projects), so `license-check` failed the post-migration build for a reason unrelated to migrating. Write the same header-only config the `test-matrix` lane uses, which also carries the exclude patterns a non-git workspace needs. Found by publishing the local build to verdaccio as a synthetic start version so the hop actually took the `internal#test-matrix` path, which no released version reaches yet.
`assertMigrationRunOutcome` scraped `nx migrate --run-migrations` output to infer what each migration did — matching a per-migration header, a success banner, and the `agentic flow disabled` notice. That tested Nx's presentation rather than our migrations, and was quietly wrong: it looked for `package:name` while Nx prints `package: name`. What matters is that the workspace still syncs and builds after migrating, which the existing assertions already cover, so the whole helper is gone along with the migration-shape interface it needed. Inline the matrix generator invocation at its call site, so it reads as what the hop scaffolds with rather than hiding behind a one-line helper. Trim the contributing guide to match: drop the `NX_E2E_MIGRATE_VERSIONS` note and the manual-testing paragraph, and stop describing assertions the test no longer makes.
nx-plugin-for-aws
force-pushed
the
feat/e2e-migrate-harness
branch
from
August 2, 2026 10:42
ccfeec6 to
ae76d7b
Compare
cogwirrel
approved these changes
Aug 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reason for this change
#928 wired up
nx migrate @aws/nx-plugin, but nothing exercises it end to end. Unit tests prove a migration does what it was written to do against a syntheticTree; they can't catch ordering problems between migrations, interactions across a whole upgrade, a migration that leaves the workspace unbuildable, or drift between what a migration writes and what today's generators produce.The contract worth defending is narrow and testable: deterministic migrations alone must keep a vanilla generated workspace green across an upgrade, non-interactively. This adds the harness that holds it.
Closes #705
Description of changes
New
migratesmoke testCreates a workspace on a released version of the plugin, scaffolds a recipe with that version's generators, then upgrades it to the local build with the real
nx migratecycle (migrate→ install →migrate --run-migrations) and asserts the result. Oneitper start version, so a failure names the hop that broke.The "before" workspace is created for real from the registry rather than from a committed fixture, so each hop starts from exactly what a user on that version has — no fixture upkeep, and no building old tags.
Per hop it asserts:
nx migratewrote the new version and generatedmigrations.json(a silent "No updates were applied" fails).tools/ai-migrations/and reported in next steps rather than silently disappearing.--run-migrationsleaves an emptygit status.syncs and builds.An empty collection is a valid outcome (that's the state on
maintoday): the test says so explicitly and still asserts the upgraded workspace is green.Registry setup (
global-setup.ts)Two things the migrate test needs that the other smoke tests don't:
nx migrateonly runs a migration whose version is greater than the installed one, so publishing the in-repo0.0.0would put every migration out of range and silently assert nothing — verified: it reports "No updates were applied". It's republished under aprereleasebump of the latest tag, withmigrations.jsonstamped byscripts/stamp-migrations.tsexactly as the release job does, under a dedicated dist-tag solatestkeeps pointing at the0.0.0build every other smoke test resolves..verdaccio/config.ymldeliberately doesn't proxy the three local packages to npmjs (our0.0.0collides with an early public release), so a released version isn't reachable through the registry the workspaces are pinned to. Mirroring the tarballs in means one@aws:registrypin serves both ends of the upgrade.A failure in this setup is recorded and surfaced by the migrate test rather than thrown, so a registry hiccup fetching an old release doesn't take down every other smoke test sharing the same setup.
Supporting changes
createTestWorkspacetakes an optionalversion, pinning@aws/create-nx-workspace(and hence the preset) to a released version. It already threaded atagargument through tobuildCreateNxWorkspaceCommandand dropped it.pinAwsScopeToLocalRegistryextracted fromsmoke-test.ts(unchanged behaviour) and reused, with anextraparameter for additional npmrc lines. The migrate test needslegacy-peer-deps— the preset is pinned to an exact version, whose peer ranges npm's strict resolver rejects duringcreate(matching whatrunInstallalready does for npm).scripts/stamp-migrations.tstakes--outso the same stamping logic writes into the staging copy, rather than duplicating it in the e2e setup.smoke_testsmatrix on both PR and CI — it's no slower than the other workspace-building smoke tests, and migration regressions are exactly what PR CI should catch.Start-version matrix
Read from
v*git tags, newest first, defaulting to the latest release — the version a user upgrading today actually starts from.NX_E2E_MIGRATE_VERSIONS=<n>widens it to the last n releases.Recipe scope
A fixed representative slice of the dungeon-adventure shape (infra, website + auth, tRPC API, FastAPI, a lambda function, an agent) rather than the full generator matrix.
This is a coupling and cost tradeoff rather than a compatibility one — checking the matrix's generators and options against past releases, it's close to compatible over the realistic window (only
ts#dcr-proxyis absent at rc.40, pluspy#rdbat rc.30, andpy#agent --framework=langchainback at rc.20; every flag name it passes still exists in all of them). The problem is that the matrix grows with every new generator, and one that postdates a start version makesnx gexit non-zero — failing the hop during scaffolding as though a migration broke. Guarding that means version-gating each of its ~60 invocations, most of which add no migration coverage (the full matrix takes ~31 minutes on theidempotencylane, and that cost multiplies per start version). The fixed recipe avoids both, and adding a project type a migration touches is one more line.Description of how you validated changes
Ran the new test locally against real migrations, using three throwaway example migrations (one of each kind, scaffolded with
ts#nx-migrationand given real bodies) published to verdaccio. These are deliberately not in this PR — they exist only to prove the harness detects what it claims to:nx migrateresolved the stamped local build, wrotemigrations.jsonwith all three entries, and materialised two prompts totools/ai-migrations/@aws/nx-plugin/<version>/(de-duplicated asprompt.md/prompt-1.md). The deterministic and hybrid codemods applied (UPDATE nx.json,UPDATE package.json); both agentic halves were skipped with "agentic flow disabled" and listed in next steps. Idempotency re-run produced an emptygit status, the customised user file survived, andrun-many --target build --allpassed.nx migratereports "There are no migrations to run", the test logs that no migrations were queued, and still asserts the upgraded workspace builds. Passes.0.0.0failure mode is real: publishing the local build at its in-repo version makesnx migratereport "No updates were applied" and run nothing — which is why the republish-at-a-higher-version step exists.biome.json; the test failed on the workspace build (formattarget rejected the config) rather than passing silently.scripts/stamp-migrations.ts --pending-version 1.0.0-rc.49stamps all unversioned entries;--outwrites to the staging path), plus the existingmigration-versions.spec.tsunit tests (19) still pass.pnpm nx run-many --target lint --allpasses.Issue # (if applicable)
Closes #705.
Checklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license