Feature request: JUnit XML converter (junit2hdf)

[clem-field]

Summary

There is no converter for JUnit XML, which is the most widely produced machine-readable test-results format across ecosystems. I would like to propose one, and to check the direction is welcome before anyone writes code.

Why this is a reasonable fit for OHDF

OHDF is increasingly used as a compliance evidence format, not only as a findings format. Two controls are evidenced by test execution rather than by scanner output:

• NIST SP 800-53r5 SA-11 — Developer Testing and Evaluation
• SSDF PW.8 — Test executable code to identify vulnerabilities and verify compliance

Today there is no path from a test suite into OHDF, so that evidence stays in CI artifacts with a retention window and never reaches Heimdall alongside the SAST, SCA and InSpec results it belongs next to.

The existing converter set already reaches beyond classic vulnerability scanners — cyclonedx-sbom is inventory, aws-config is configuration compliance, msft-secure-score is a posture score. Test results feel like a smaller step than any of those.

Why JUnit XML specifically

It is the closest thing the industry has to a universal test-results interchange format, emitted natively or via one flag by pytest (--junitxml), Playwright, Jest, Maven/Gradle, Go (gotestsum), RSpec (formatter), PHPUnit and .NET (via trx conversion). One converter covers most languages, which is not true of the per-vendor mappers.

For comparison, jfrog-xray-mapper.ts serves a single commercial product. JUnit XML serves nearly every test runner in use.

The mapping is unusually close to HDF's own shape

JUnit XML is testsuite → testcase, and HDF is control → results[]. That suggests a natural, low-invention mapping:

JUnit	HDF
<testsuite name>	control id / title
<testcase>	one entry in results[]
<testcase> with no child	status: passed
<failure> / <error>	status: failed, message into message
<skipped>	status: skipped, into skip_message
time attribute	run_time
classname + name	code_desc

Suite-level granularity matters: mapping each test case to its own control would produce thousands of controls for a normal suite and make Heimdall unreadable. Mapping the suite to a control and its cases to results keeps it proportionate, and matches how HDF already represents a control checked several ways.

Open questions worth deciding before implementation

1. Impact. Test results carry no severity. A fixed default (0.5?) with a CLI override seems least surprising, but there may be an existing convention I have missed.
2. Tags. JUnit carries no control mappings. Allowing caller-supplied tags (--tags nist=SA-11) would let the evidence land against the right control without the converter inventing one.
3. Nested suites. Some producers emit <testsuites> wrapping several <testsuite> elements; flattening versus one control per inner suite is a judgement call.
4. Scope. Should a clean run with zero failures convert at all? Our position is yes — a suite that ran and passed is exactly the evidence that is hardest to produce after the fact — but it does mean an all-passing OHDF file, which is unusual for this toolchain.

Context

We hit this building a DevSecOps evidence profile: the pipeline runs pytest, Playwright and RSpec, all of which already emit JUnit XML, and none of which can reach the evidence store. We have written a local mapper as a stopgap and would be glad to contribute a proper converter upstream if the direction is welcome — happy to follow whatever shape you prefer for the open questions above.

[clem-field]

Settling granularity before this gets written

The proposal above maps suite to control, case to result, on the grounds
that per-case controls would produce thousands of controls and make Heimdall
unreadable. That reasoning is sound on readability but I think it has to be
reversed, for a reason that only shows up in scoring.

inspecjs derives a control's status from its results with "any failed tests
→ Failed" (libs/inspecjs/src/compat_wrappers.ts, rule 3 in the enumeration
at the top of the file).

So under suite-to-control, a suite of 1,000 tests with a single failure converts
to one control, status Failed. The 999 passing cases contribute nothing —
they are results inside a control that has already been decided by the one
failure. A 99.9% passing suite renders as 100% failed.

That is not a display quirk. It is the compliance score, and it is the number
someone reads to decide whether SA-11 is satisfied.

Under case-to-control the same suite converts to 1,000 controls, 999 Passed and
1 Failed, and the score reflects what actually happened.

The other consideration

junit-to-hdf already ships in mitre/hdf-libs and maps one requirement per
test case (testCaseToRequirement). Whatever this converter does, the two
lines should agree for the same input format — a document converting to a
different shape depending on which CLI a user reached for is worse than either
choice on its own merits.

On the readability objection

It is real but I think overstated. Profiles of that size are ordinary: the NIST
SP 800-53 Rev 5 profile is 1,014 controls, and STIG baselines run to several
hundred. Heimdall handles them.

The mitigation is to keep the suite identity queryable rather than to collapse
the structure. junit-to-hdf currently puts classname into the requirement
description but does not carry the suite name as a tag; adding
tags.testSuite on both sides would let Heimdall group and filter by suite,
which recovers the navigability without costing the score.

Proposed revision to this issue

JUnit	HDF
<testcase>	one control
<testsuite name>	tags.testSuite on each control it contains
<testcase classname> + name	control id / title
no child element	status: passed
<failure> / <error>	status: failed, body into message
<skipped>	status: skipped, into skip_message
time	run_time

The suite still appears in the output; it just stops being the unit that decides
pass or fail.

Happy to be argued out of this if there is a scoring behaviour I have missed —
the whole argument rests on that one rule in compat_wrappers.ts.

Related: mitre/hdf-libs#241, which raises a separate JUnit question — Checkov can
emit JUnit XML, and it currently routes to the generic converter and loses its
metadata.