Skip to content

No build or test CI on pull requests, and the test suite never runs in CI #495

Description

@davidpoblador

Problem

Nothing builds or tests this repository on a pull request, and the test suite never runs in CI at all.

Current pull_request triggers:

Workflow Trigger What it does
claude-code-review.yml pull_request AI review, no build
pr-title-check.yml pull_request_target conventional-commit title lint

And the rest:

Workflow Trigger Builds? Tests?
release.yml push to main, dispatch yes no
test-ghostty.yml workflow_dispatch only yes yes
deploy-website.yml push to main (website/**), dispatch site only no

So:

  • A PR can be merged without anything ever compiling it.
  • release.yml builds but has no test step, so a green release does not imply passing tests.
  • The XCTest suite (209 tests) only runs via test-ghostty.yml, which is manual-dispatch only and exists to check compatibility against a new Ghostty tag, not to gate changes.
  • The Python script tests under scripts/test_*.py (18 tests covering changelog.py and appcast generation) run nowhere in CI.

Why it matters

The first time a change is compiled by CI is during the release build, on main, after merge. A compile error or failing test is therefore discovered at release time, and recovery means shipping another version rather than pushing a commit to a branch.

This is not hypothetical. 0.1.77 was tagged and published with no downloadable artifact because its release build failed, and 0.1.78 had to be cut to replace it. That particular failure was a bad submodule entry that any PR-level checkout would have caught immediately.

It also means correctness currently rests on whoever wrote the change having run ./scripts/dev.sh test locally, with nothing recording whether they did.

Proposal

Add a pull_request workflow that builds and tests. test-ghostty.yml already contains a working recipe (checkout with submodules, brew install xcodegen, ./.github/actions/install-zig, setup-bun, zig build the xcframework, xcodegen generate, then the FactoryFloorTests scheme).

Points worth deciding:

  • Cost and time. macos-15 minutes are expensive. The ghostty xcframework build dominates, but release.yml already caches it on ghostty-build-${{ hashFiles('ghostty/.git') }}, and caches created on main are readable from PR branches, so most PRs should hit it. Building -Dxcframework-target=aarch64-macos rather than universal would also cut it down for a test-only build.
  • Scope. Could be limited with paths to Sources/**, Tests/**, project.yml, Resources/** so documentation and website PRs stay fast, at the cost of the filter drifting out of date.
  • Add a test step to release.yml. Independently useful, so a release cannot be cut from a tree whose tests fail.
  • Run the Python tests. uvx pytest scripts/ on ubuntu-latest costs almost nothing and currently has no coverage at all.
  • Formatting. prek hooks (SwiftFormat, trailing whitespace, YAML/JSON checks) run locally on commit but are not enforced in CI, so a contributor without hooks installed can land unformatted code.

Happy to implement whichever combination you prefer.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions