Repository navigation
Linter: Add --format junit for CI test reports - #2748
Draft
pinzonjulian wants to merge 4 commits into
Draft
pinzonjulian wants to merge 4 commits into
pinzonjulian wants to merge 4 commits into
Conversation
CI runs want readable output in the job log and a structured report for tooling to parse and store, without linting twice. `--format` can now repeat, and `--output-file` writes the json format before it to a file, following RuboCop's --format/--out pairing. Reports are written after the terminal output, a file that can't be written fails the run without losing the other outputs, and invalid configs or patterns without matches still produce a report. When several formats are left on stdout, the existing --json > --simple > --format precedence applies. Amp-Thread-ID: https://ampcode.com/threads/T-01a0f508-1d83-7209-a789-c4c2314f0299 Co-authored-by: Amp <amp@ampcode.com>
CI systems like Buildkite Test Engine, GitLab and Jenkins ingest JUnit XML as test results, so reporting offenses that way surfaces them as failing tests. It works on stdout and with --output-file. Each file is a testsuite and each rule with offenses a testcase, which keeps test identities stable when lines move. A testcase fails at the --fail-level, including offenses --log-level hides, so the report agrees with the exit code. Early failures render an erroring testcase and a run with nothing to lint a skipped one, since Jenkins rejects empty reports. Since XML breaks on anything else printed to stdout, --force notices and stimulus-lint's project analysis move to stderr, and GitHub Actions annotations detected from the environment are skipped for structured formats instead of failing the run; only an explicit --github errors. Amp-Thread-ID: https://ampcode.com/threads/T-01a0f508-1d83-7209-a789-c4c2314f0299 Co-authored-by: Amp <amp@ampcode.com>
Review feedback: declaring the list as const and inferring the type keeps them from drifting, following DIAGNOSTIC_SEVERITIES. Also badges the Multiple Outputs section with its release. Amp-Thread-ID: https://ampcode.com/threads/T-01a0f508-1d83-7209-a789-c4c2314f0299 Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a0f508-1d83-7209-a789-c4c2314f0299 Co-authored-by: Amp <amp@ampcode.com> # Conflicts: # javascript/packages/linter/README.md # javascript/packages/linter/src/cli/argument-parser.ts
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request adds a
junitoutput format toherb-lint, so CI systems can show offenses as failing tests.JUnit XML is what most CI systems read test results from: GitLab (
artifacts:reports:junit), Jenkins, CircleCI (store_test_results), Azure Pipelines (PublishTestResults), Buildkite Test Engine and Buildkite'sjunit-annotateplugin. GitHub Actions has no native JUnit support, which--githubalready covers. On the other systems, a lint failure today is a red step and a log to scroll through, rather than a list of failures the CI tooling can display, group and track.It is also a gap for teams moving from
erb_lint, which has had--format junitsince 0.5.0. RuboCop (since 0.80), Brakeman, Biome, Oxlint, Ruff, Pylint and golangci-lint all ship JUnit output, and ESLint moved it toeslint-formatter-junitin v9.With
--output-file, the job log keeps the readable output while the report goes to the CI system:Each linted file becomes a
<testsuite>, and each rule with offenses in that file becomes a<testcase>with the file as itsclassnameand the rule as itsname. Grouping by rule rather than by offense keeps a test's identity stable when lines move, which matters to tools that follow a test across builds. A file without offenses gets one passing testcase. The trade-off is that fixing a rule's last offense removes its testcase instead of turning it green.A testcase fails when one of its offenses meets
--fail-level. Offenses below it are listed in the<system-out>of a passing testcase. Offenses that fail the run are included even when--log-levelhides them, so the report fails exactly whenherb-lintexits with an error.A run that stops before linting, such as an invalid configuration, a pattern without matching files or an unknown rule passed to
--only, produces one erroring testcase with the message. A run with nothing to lint, such as a disabled linter, produces one skipped testcase, because Jenkins rejects reports without testcases by default.Anything else printed to stdout would break the XML, so a few changes also apply to
--json:--forcenotices and stimulus-lint's project analysis messages now go to stderr.GITHUB_ACTIONS=trueused to make--jsonfail on every GitHub Actions runner unless--no-githubwas passed. Now only an explicit--githubis rejected, and annotations detected from the environment are skipped when JSON or JUnit is printed to stdout.The GitLab CI docs gain a test report example.