Skip to content

test(e2e): migrate a released workspace to the local build - #1002

Merged
cogwirrel merged 8 commits into
mainfrom
feat/e2e-migrate-harness
Aug 2, 2026
Merged

test(e2e): migrate a released workspace to the local build#1002
cogwirrel merged 8 commits into
mainfrom
feat/e2e-migrate-harness

Conversation

@nx-plugin-for-aws

@nx-plugin-for-aws nx-plugin-for-aws commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Reason for this change

#928 wired up nx migrate @aws/nx-plugin, but nothing exercises it end to end. Unit tests prove a migration does what it was written to do against a synthetic Tree; they can't catch ordering problems between migrations, interactions across a whole upgrade, a migration that leaves the workspace unbuildable, or drift between what a migration writes and what today's generators produce.

The contract worth defending is narrow and testable: deterministic migrations alone must keep a vanilla generated workspace green across an upgrade, non-interactively. This adds the harness that holds it.

Closes #705

Description of changes

New migrate smoke test

Creates a workspace on a released version of the plugin, scaffolds a recipe with that version's generators, then upgrades it to the local build with the real nx migrate cycle (migrate → install → migrate --run-migrations) and asserts the result. One it per start version, so a failure names the hop that broke.

The "before" workspace is created for real from the registry rather than from a committed fixture, so each hop starts from exactly what a user on that version has — no fixture upkeep, and no building old tags.

Per hop it asserts:

  • The upgrade did somethingnx migrate wrote the new version and generated migrations.json (a silent "No updates were applied" fails).
  • The installed plugin is the target — so the migrations that run are the ones the target version ships, not the ones already installed.
  • Codemods ran — every queued entry appears in the run output; a throwing codemod fails the command.
  • Prompts degrade correctly — prompt and hybrid migrations are materialised under tools/ai-migrations/ and reported in next steps rather than silently disappearing.
  • User code is safe — a deliberately customised user-owned file must survive verbatim (guardrail: report what you don't recognise, don't rewrite it).
  • Idempotency — re-running --run-migrations leaves an empty git status.
  • The core contract — the migrated workspace still syncs and builds.

An empty collection is a valid outcome (that's the state on main today): the test says so explicitly and still asserts the upgraded workspace is green.

Registry setup (global-setup.ts)

Two things the migrate test needs that the other smoke tests don't:

  • The local build is republished at a version above every release. nx migrate only runs a migration whose version is greater than the installed one, so publishing the in-repo 0.0.0 would put every migration out of range and silently assert nothing — verified: it reports "No updates were applied". It's republished under a prerelease bump of the latest tag, with migrations.json stamped by scripts/stamp-migrations.ts exactly as the release job does, under a dedicated dist-tag so latest keeps pointing at the 0.0.0 build every other smoke test resolves.
  • Each start version is mirrored into verdaccio. .verdaccio/config.yml deliberately doesn't proxy the three local packages to npmjs (our 0.0.0 collides with an early public release), so a released version isn't reachable through the registry the workspaces are pinned to. Mirroring the tarballs in means one @aws:registry pin serves both ends of the upgrade.

A failure in this setup is recorded and surfaced by the migrate test rather than thrown, so a registry hiccup fetching an old release doesn't take down every other smoke test sharing the same setup.

Supporting changes

  • createTestWorkspace takes an optional version, pinning @aws/create-nx-workspace (and hence the preset) to a released version. It already threaded a tag argument through to buildCreateNxWorkspaceCommand and dropped it.
  • pinAwsScopeToLocalRegistry extracted from smoke-test.ts (unchanged behaviour) and reused, with an extra parameter for additional npmrc lines. The migrate test needs legacy-peer-deps — the preset is pinned to an exact version, whose peer ranges npm's strict resolver rejects during create (matching what runInstall already does for npm).
  • scripts/stamp-migrations.ts takes --out so the same stamping logic writes into the staging copy, rather than duplicating it in the e2e setup.
  • Runs in the existing smoke_tests matrix on both PR and CI — it's no slower than the other workspace-building smoke tests, and migration regressions are exactly what PR CI should catch.

Start-version matrix

Read from v* git tags, newest first, defaulting to the latest release — the version a user upgrading today actually starts from. NX_E2E_MIGRATE_VERSIONS=<n> widens it to the last n releases.

Recipe scope

A fixed representative slice of the dungeon-adventure shape (infra, website + auth, tRPC API, FastAPI, a lambda function, an agent) rather than the full generator matrix.

This is a coupling and cost tradeoff rather than a compatibility one — checking the matrix's generators and options against past releases, it's close to compatible over the realistic window (only ts#dcr-proxy is absent at rc.40, plus py#rdb at rc.30, and py#agent --framework=langchain back at rc.20; every flag name it passes still exists in all of them). The problem is that the matrix grows with every new generator, and one that postdates a start version makes nx g exit non-zero — failing the hop during scaffolding as though a migration broke. Guarding that means version-gating each of its ~60 invocations, most of which add no migration coverage (the full matrix takes ~31 minutes on the idempotency lane, and that cost multiplies per start version). The fixed recipe avoids both, and adding a project type a migration touches is one more line.

Description of how you validated changes

Ran the new test locally against real migrations, using three throwaway example migrations (one of each kind, scaffolded with ts#nx-migration and given real bodies) published to verdaccio. These are deliberately not in this PR — they exist only to prove the harness detects what it claims to:

  • All three kinds queued and handled: nx migrate resolved the stamped local build, wrote migrations.json with all three entries, and materialised two prompts to tools/ai-migrations/@aws/nx-plugin/<version>/ (de-duplicated as prompt.md / prompt-1.md). The deterministic and hybrid codemods applied (UPDATE nx.json, UPDATE package.json); both agentic halves were skipped with "agentic flow disabled" and listed in next steps. Idempotency re-run produced an empty git status, the customised user file survived, and run-many --target build --all passed.
  • With the empty collection this PR ships: nx migrate reports "There are no migrations to run", the test logs that no migrations were queued, and still asserts the upgraded workspace builds. Passes.
  • The 0.0.0 failure mode is real: publishing the local build at its in-repo version makes nx migrate report "No updates were applied" and run nothing — which is why the republish-at-a-higher-version step exists.
  • The harness catches a broken migration: an early version of the example hybrid migration wrote an invalid key into biome.json; the test failed on the workspace build (format target rejected the config) rather than passing silently.
  • Stamping verified against real tag history (scripts/stamp-migrations.ts --pending-version 1.0.0-rc.49 stamps all unversioned entries; --out writes to the staging path), plus the existing migration-versions.spec.ts unit tests (19) still pass.
  • pnpm nx run-many --target lint --all passes.

Issue # (if applicable)

Closes #705.

Checklist


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@codecov-commenter

codecov-commenter commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.59%. Comparing base (76225dd) to head (ae76d7b).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1002   +/-   ##
=======================================
  Coverage   88.59%   88.59%           
=======================================
  Files         180      180           
  Lines        6699     6699           
  Branches     1622     1622           
=======================================
  Hits         5935     5935           
  Misses        364      364           
  Partials      400      400           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@nx-plugin-for-aws
nx-plugin-for-aws force-pushed the feat/e2e-migrate-harness branch from 0c1894c to aa9a245 Compare July 31, 2026 04:45
Comment thread e2e/src/smoke-tests/migrate.ts Outdated
Comment thread e2e/src/smoke-tests/migrate.ts Outdated
Comment thread e2e/src/smoke-tests/migrate.ts Outdated
Comment thread e2e/src/global-setup.ts Outdated
Comment thread e2e/src/smoke-tests/migrate-versions.ts Outdated
@nx-plugin-for-aws
nx-plugin-for-aws force-pushed the feat/e2e-migrate-harness branch from 807feb2 to 0353eb2 Compare August 1, 2026 05:20
nx-plugin-for-aws and others added 8 commits August 2, 2026 10:14
Adds a `migrate` smoke test that upgrades a workspace generated by a released
version of the plugin to the local build using the real `nx migrate` cycle, so
the migration collection is regressed against a workspace a user actually has
rather than only against synthetic trees in unit tests.

Closes #705
- Run the hops under pnpm rather than npm, relaxing peer strictness with
  pnpm's own `strict-peer-dependencies=false` instead of `legacy-peer-deps`.
- Scaffold with `internal#test-matrix` so each hop gets the start version's own
  full generator coverage. Releases predating that generator keep the fixed
  recipe; the fallback goes when the supported range no longer reaches them.
- Migrate from the last 5 releases rather than only the latest.
- Move `publishForMigrateSmokeTest` into `migrate-versions`, alongside the
  version resolution it depends on, and skip it unless the run includes the
  migrate test — every lane shares global-setup, and mirroring a release per
  start version is wasted work for the lanes that never read it.
- Drop the user-customisation assertion, which hardcoded a path from the
  recipe's website project and kept the driver from staying generic.

Assert the upgraded version against the installed package rather than the
manifest range: on a pnpm workspace the manifest holds `catalog:` and the
version lives in `pnpm-workspace.yaml`, so node_modules is the
package-manager-agnostic record — and what the migrations actually run from.
Replace the hardcoded `FIRST_VERSION_WITH_TEST_MATRIX` gate with a runtime
capability check: the hop runs `nx list @aws/nx-plugin` against the workspace it
just created and looks for `internal#test-matrix`. No release boundary is
encoded, and the answer comes from the collection the hop is about to generate
from. A start version that does not vend it is skipped with a warning rather
than falling back to a hand-maintained recipe, so the fixed recipe is gone.

Verified the check by publishing two plugin builds to a local verdaccio, one
with `internal#test-matrix` and one with it stripped, then running the predicate
against a workspace on each: true and false respectively. Also verified against
a real 1.0.0-rc.50 workspace built from the published tarball, which correctly
reports false.

Also assert the "before" workspace is actually on the start version before
migrating. `nx migrate` only runs migrations above the installed version, so a
workspace that silently resolved the local `0.0.0` build (published to verdaccio
as `latest`) yields a hop that migrates nothing and still passes. This assertion
fires today, which is a pre-existing hole in the harness rather than a
regression: the log shows create selecting `preset=@aws/nx-plugin@1.0.0-rc.50`
while the resulting catalog pins `0.0.0`, so the executed preset was the local
build. Left failing deliberately rather than asserting a hop that proves nothing.
`@nx/workspace`'s new generator resolves the preset's version as
`NX_E2E_PRESET_VERSION ?? <version parsed from --preset>` — the environment
variable takes priority. `global-setup` sets it to the local build's version so
the other smoke tests install that, which silently overrode the version pin the
migrate test passes: `create` reported `preset=@aws/nx-plugin@1.0.0-rc.50` while
the workspace was scaffolded and installed at `0.0.0`.

Because `nx migrate` only runs migrations whose version is above the installed
one, every hop was really `0.0.0 -> target`, leaving every migration out of range
and asserting nothing about a real upgrade. The lane was green because it was
vacuous.

`createTestWorkspace` now points `NX_E2E_PRESET_VERSION` at the version it was
asked for, so the variable can no longer contradict the pin. Only the migrate
test passes a version, so no other lane changes behaviour.

Verified end to end: the workspace now installs 1.0.0-rc.50, which the assertion
added alongside this confirms — it failed before this change and passes after.
Until a release ships `internal#test-matrix`, every hop skips itself and the lane
passes having asserted nothing. Log which start versions migrated and how many
skipped, and say explicitly when none ran, so a green tick can't imply coverage
that isn't there.
The matrix runs the license generator, whose dependency allowlist rejects some of
what the matrix itself pulls in (`mariadb` is LGPL-2.1-or-later, via the mysql
rdb projects), so `license-check` failed the post-migration build for a reason
unrelated to migrating. Write the same header-only config the `test-matrix` lane
uses, which also carries the exclude patterns a non-git workspace needs.

Found by publishing the local build to verdaccio as a synthetic start version so
the hop actually took the `internal#test-matrix` path, which no released version
reaches yet.
`assertMigrationRunOutcome` scraped `nx migrate --run-migrations` output to infer
what each migration did — matching a per-migration header, a success banner, and
the `agentic flow disabled` notice. That tested Nx's presentation rather than our
migrations, and was quietly wrong: it looked for `package:name` while Nx prints
`package: name`. What matters is that the workspace still syncs and builds after
migrating, which the existing assertions already cover, so the whole helper is
gone along with the migration-shape interface it needed.

Inline the matrix generator invocation at its call site, so it reads as what the
hop scaffolds with rather than hiding behind a one-line helper.

Trim the contributing guide to match: drop the `NX_E2E_MIGRATE_VERSIONS` note and
the manual-testing paragraph, and stop describing assertions the test no longer
makes.
@nx-plugin-for-aws
nx-plugin-for-aws force-pushed the feat/e2e-migrate-harness branch from ccfeec6 to ae76d7b Compare August 2, 2026 10:42
@cogwirrel
cogwirrel merged commit 60e3444 into main Aug 2, 2026
58 checks passed
@cogwirrel
cogwirrel deleted the feat/e2e-migrate-harness branch August 2, 2026 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: E2E test harness for rolling migrations between versions

3 participants