Skip to content

Linter: Add --format junit for CI test reports - #2748

Draft
pinzonjulian wants to merge 4 commits into
marcoroth:mainfrom
pinzonjulian:linter-junit
Draft

pinzonjulian wants to merge 4 commits into
marcoroth:mainfrom
pinzonjulian:linter-junit

Conversation

@pinzonjulian

Copy link
Copy Markdown
Contributor

Builds on #OUTPUT_FILE_PR, which adds --output-file. Its commit is included here, so only the last commit is new in this pull request.

This pull request adds a junit output format to herb-lint, so CI systems can show offenses as failing tests.

JUnit XML is what most CI systems read test results from: GitLab (artifacts:reports:junit), Jenkins, CircleCI (store_test_results), Azure Pipelines (PublishTestResults), Buildkite Test Engine and Buildkite's junit-annotate plugin. GitHub Actions has no native JUnit support, which --github already covers. On the other systems, a lint failure today is a red step and a log to scroll through, rather than a list of failures the CI tooling can display, group and track.

It is also a gap for teams moving from erb_lint, which has had --format junit since 0.5.0. RuboCop (since 0.80), Brakeman, Biome, Oxlint, Ruff, Pylint and golangci-lint all ship JUnit output, and ESLint moved it to eslint-formatter-junit in v9.

With --output-file, the job log keeps the readable output while the report goes to the CI system:

herb-lint --format detailed --format junit -o herb-lint.xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="herb-lint" tests="2" failures="1" errors="0" time="0.312">
  <testsuite name="app/views/home.html.erb" tests="2" failures="1" errors="0" skipped="0" time="0">
    <testcase classname="app/views/home.html.erb" name="html-img-require-alt" file="app/views/home.html.erb" line="3" time="0">
      <system-out>app/views/home.html.erb:3:3: warning: Missing required `alt` attribute on `&lt;img&gt;` tag. ...</system-out>
    </testcase>
    <testcase classname="app/views/home.html.erb" name="html-tag-name-lowercase" file="app/views/home.html.erb" line="2" time="0">
      <failure message="Opening tag name `&lt;SPAN&gt;` should be lowercase. Use `&lt;span&gt;` instead. (and 1 more)" type="error">app/views/home.html.erb:2:3: error: Opening tag name `&lt;SPAN&gt;` should be lowercase. Use `&lt;span&gt;` instead.
app/views/home.html.erb:2:22: error: Closing tag name `&lt;/SPAN&gt;` should be lowercase. Use `&lt;/span&gt;` instead.

https://herb-tools.dev/linter/rules/html-tag-name-lowercase</failure>
    </testcase>
  </testsuite>
</testsuites>

Each linted file becomes a <testsuite>, and each rule with offenses in that file becomes a <testcase> with the file as its classname and the rule as its name. Grouping by rule rather than by offense keeps a test's identity stable when lines move, which matters to tools that follow a test across builds. A file without offenses gets one passing testcase. The trade-off is that fixing a rule's last offense removes its testcase instead of turning it green.

A testcase fails when one of its offenses meets --fail-level. Offenses below it are listed in the <system-out> of a passing testcase. Offenses that fail the run are included even when --log-level hides them, so the report fails exactly when herb-lint exits with an error.

A run that stops before linting, such as an invalid configuration, a pattern without matching files or an unknown rule passed to --only, produces one erroring testcase with the message. A run with nothing to lint, such as a disabled linter, produces one skipped testcase, because Jenkins rejects reports without testcases by default.

Anything else printed to stdout would break the XML, so a few changes also apply to --json:

  • The --force notices and stimulus-lint's project analysis messages now go to stderr.
  • GITHUB_ACTIONS=true used to make --json fail on every GitHub Actions runner unless --no-github was passed. Now only an explicit --github is rejected, and annotations detected from the environment are skipped when JSON or JUnit is printed to stdout.

The GitLab CI docs gain a test report example.

pinzonjulian and others added 2 commits October 1, 2026 03:00
CI runs want readable output in the job log and a structured report for
tooling to parse and store, without linting twice. `--format` can now
repeat, and `--output-file` writes the json format before it to a file,
following RuboCop's --format/--out pairing.

Reports are written after the terminal output, a file that can't be
written fails the run without losing the other outputs, and invalid
configs or patterns without matches still produce a report. When several
formats are left on stdout, the existing --json > --simple > --format
precedence applies.

Amp-Thread-ID: https://ampcode.com/threads/T-01a0f508-1d83-7209-a789-c4c2314f0299
Co-authored-by: Amp <amp@ampcode.com>
CI systems like Buildkite Test Engine, GitLab and Jenkins ingest JUnit XML
as test results, so reporting offenses that way surfaces them as failing
tests. It works on stdout and with --output-file.

Each file is a testsuite and each rule with offenses a testcase, which
keeps test identities stable when lines move. A testcase fails at the
--fail-level, including offenses --log-level hides, so the report agrees
with the exit code. Early failures render an erroring testcase and a run
with nothing to lint a skipped one, since Jenkins rejects empty reports.

Since XML breaks on anything else printed to stdout, --force notices and
stimulus-lint's project analysis move to stderr, and GitHub Actions
annotations detected from the environment are skipped for structured
formats instead of failing the run; only an explicit --github errors.

Amp-Thread-ID: https://ampcode.com/threads/T-01a0f508-1d83-7209-a789-c4c2314f0299
Co-authored-by: Amp <amp@ampcode.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation linter @herb-tools/linter for HTML+ERB templates typescript TypeScript source across the javascript/ packages stimulus-lint stimulus-lint rules for Stimulus controllers and view templates labels Oct 1, 2026
pinzonjulian and others added 2 commits October 6, 2026 05:39
Review feedback: declaring the list as const and inferring the type keeps them from drifting, following DIAGNOSTIC_SEVERITIES. Also badges the Multiple Outputs section with its release.

Amp-Thread-ID: https://ampcode.com/threads/T-01a0f508-1d83-7209-a789-c4c2314f0299
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a0f508-1d83-7209-a789-c4c2314f0299
Co-authored-by: Amp <amp@ampcode.com>

# Conflicts:
#	javascript/packages/linter/README.md
#	javascript/packages/linter/src/cli/argument-parser.ts

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation linter @herb-tools/linter for HTML+ERB templates stimulus-lint stimulus-lint rules for Stimulus controllers and view templates typescript TypeScript source across the javascript/ packages

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant