Skip to content

CI: Playwright e2e suite has been broken since at least 2026-05-11 #190

@devartifex

Description

@devartifex

Symptom

Every CI run on this repo for the past 2+ weeks has been failure or cancelled due to the CI/e2e job. This blocks every PR and every dependabot bump.

Evidence

gh run list --workflow=CI --limit 20 shows an unbroken streak of failures since at least 2026-05-11 across:

Failure pattern

In every run the Playwright step shows the same shape: tests start, then a flood of ××F (retry-retry-fail) markers, then the job hits the GitHub Actions timeout-minutes limit and is cancelled.

PR #189 bumped timeout-minutes from 15 → 30 to rule out budget; the suite still timed out at 30 min with the same pattern, so the cause is real test failures (not just slowness). The suite runs 126 tests with workers: 1 and retries: 2 in CI per playwright.config.ts.

Root cause is unknown

This needs dedicated investigation outside the scope of any single feature PR. Candidates:

  • Real product regression somewhere between 2026-05-11 and now
  • Server startup / shutdown race against early test requests (some failures show socket hang up against the local dev server)
  • Resource pressure with 1-worker × 2-retries × 126-test config
  • A specific spec file that hangs, blocking the rest

Suggested next steps

  1. Reproduce locally with CI=1 npx playwright test --project=desktop --reporter=list and capture which tests are flaky vs hard-failing
  2. Triage by spec file — disable the slowest/flakiest spec only after attaching a tracking link
  3. Consider raising workers on CI (the bottleneck is wall clock, not CPU contention)
  4. Once green, set retries: 0 on the worst offenders so flake stays visible

Out of scope for this issue

Do not paper over the failure by skipping the e2e job, marking it optional, or removing tests. Fix the underlying problem.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions