Skip to content

feat(hdf-converters): add Semgrep JSON to HDF mapper - #8571

Open
clem-field wants to merge 1 commit into
mitre:masterfrom
clem-field:feature/semgrep2hdf
Open

feat(hdf-converters): add Semgrep JSON to HDF mapper#8571
clem-field wants to merge 1 commit into
mitre:masterfrom
clem-field:feature/semgrep2hdf

Conversation

@clem-field

Copy link
Copy Markdown
Contributor

Adds a mapper for the native semgrep scan --json format, plus its fingerprint, frontend intake wiring, sample data and spec.

Why not just use sarif2hdf

Semgrep emits SARIF, so sarif2hdf covers it in principle. But SARIF carries Semgrep's rule metadata only as untyped prose tags on the rule object ("CWE-939: ...", "LOW CONFIDENCE") and drops the rest outright: impact, likelihood, the ASVS control mapping, references[], vulnerability_class, technology and subcategory.

Those are the cross-framework links that make results usable for accreditation — the ASVS control_id in particular — so the native format gets its own mapper.

Granularity

One control per rule. Semgrep metadata is rule-scoped and identical across occurrences, so only the location varies and occurrences collapse into results under a single control keyed on check_id.

Format notes

All of these were confirmed against semgrep 1.174.0 output rather than its documentation, which is incomplete on each point:

  • metadata fields documented as arrays arrive as bare strings when a rule declares a single value (owasp, subcategory, technology, references); everything list-shaped is normalized on read
  • CWEs are emitted in prose form ("CWE-89: Improper Neutralization of ..."), so the id is parsed out before the NIST lookup
  • extra.lines and extra.fingerprint are redacted to the literal string "requires login" in unauthenticated scans, and are filtered rather than mapped
  • metadata.impact rates the severity of the consequence and is not HDF's impact float; it is tagged semgrep_impact so it cannot shadow it
  • findings suppressed with a nosemgrep comment are omitted from the output entirely rather than flagged, so no skipped status is derivable and every finding is reported as failed
  • extra.fix is replacement text for the matched span, not a standalone instruction — rendering it bare produces messages like Suggested fix: False
  • the JSON output carries no human-readable rule title anywhere, unlike the SARIF output whose rule objects have name and shortDescription, so a title is derived from the final segment of the dotted rule id
  • errors[].type is a heterogeneous array (discriminant plus optional payload) and is read only for its discriminant

Scan errors

Scan failures become their own control with status error, present only when the scan reported any. A file that failed to parse was not fully analyzed, so absence of findings in it is not evidence of compliance — burying that in passthrough makes it invisible in Heimdall.

Fingerprint

Keys on semgrep-specific paths (paths.scanned, engine_requested, skipped_rules) rather than the generic results/errors/version triple. The generic form matched bare arrays of HDF controls and would have hijacked input belonging to other converters; verified against all 135 sample files in sample_jsons/, with no collisions in either direction.

Testing

13 tests covering rule grouping, severity mapping, CWE-derived NIST tags, the string-or-array owasp normalization, the semgrep_impact rename, refs shape, the redacted-placeholder filter, the empty scan, and the scan-errors control.

vitest run in libs/hdf-converters: 167 passing. The 4 failures (3 sonarqube, 1 splunk reverse) reproduce identically on an unmodified master checkout.

Semgrep's native JSON carries a rule metadata layer that its SARIF output
drops: impact and likelihood, the ASVS control mapping, reference URLs,
vulnerability_class, technology and subcategory, and the bandit rule
cross-reference. SARIF keeps CWE, OWASP and confidence only as untyped prose
tags on the rule object. Converting the native format preserves the
cross-framework links that make the results usable for accreditation.

One control per rule: Semgrep metadata is rule-scoped and identical across
occurrences, so findings collapse into results under a single control keyed
on check_id.

Notes on the format, all confirmed against semgrep 1.174.0 output rather
than its documentation:

- metadata fields documented as arrays arrive as bare strings when a rule
  declares a single value (owasp, subcategory, technology); everything
  list-shaped is normalized on read
- CWEs are emitted in prose form ('CWE-89: Improper Neutralization of ...'),
  so the id is parsed out before the NIST lookup
- extra.lines and extra.fingerprint are redacted to the literal string
  'requires login' in unauthenticated scans and are filtered rather than
  mapped
- metadata.impact rates the severity of the consequence and is not HDF's
  impact float; it is tagged as semgrep_impact so it cannot shadow it
- findings suppressed with a nosemgrep comment are omitted from the output
  entirely rather than flagged, so no skipped status is derivable
- extra.fix is replacement text for the matched span, not a standalone
  instruction, and is labelled accordingly

Scan errors become their own control with status error, present only when
the scan produced any, so a file that failed to parse is visible rather
than buried in passthrough.

The fingerprint keys on semgrep-specific paths (paths.scanned,
engine_requested, skipped_rules) rather than the generic results/errors/
version triple, which matched bare arrays of HDF controls belonging to
other converters.

Also normalizes quoting on the adjacent CHECKOV fingerprint line to satisfy
the repository prettier configuration.

Signed-off-by: clem-field <kc8yhe@me.com>
@clem-field
clem-field force-pushed the feature/semgrep2hdf branch from db13078 to 98bff5f Compare August 23, 2026 12:41
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
1 New Bugs (required ≤ 0)
2 New Code Smells (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@mergify

mergify Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

This pull request has a conflict. Could you fix it @clem-field?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant