Skip to content

audit: warn when action_shas present and --resolve is off #57

@Fieldnote-Echo

Description

UX: warn when action_shas is present and --resolve is off

Problem

nboot audit runs offline by default — --resolve is off, so action
SHAs are left as placeholder values in the rendered plan. This is
intentional and documented (audit.md "By default audit runs
offline …"; the CLI --help says the same), so this isn't a hidden
bug.

The improvement opportunity: when a pack declares action_shas AND
the user runs offline, compute_diffs will report every affected
workflow as drifted because the rendered placeholders don't match
the on-disk real SHAs. The user has no on-screen cue that the
finding count would collapse if they passed --resolve. They then
either (a) treat the diff as a real drift list and waste triage
effort, or (b) eventually re-read the docs and re-run.

This is a UX gap, not a defect — the docs already cover it, but the
runtime experience doesn't reinforce it at the moment of confusion.

Proposed resolution

Single approach: smart warning at audit time.

run_audit() already has both skip_resolve and the loaded manifest
in scope at audit.py:133-136. When manifest.get("action_shas") is
non-empty and skip_resolve=True, emit a one-line note to stderr
before running compute_diffs:

Notice: --resolve is off and pack '<name>' declares N action SHA(s).
Workflow files rendered with placeholder SHAs may show as drift even
when the target is fully conforming. Re-run with --resolve (requires
gh) to compare against real SHAs.

Don't fail; don't suppress findings; let the user decide.

Pair with a one-paragraph addition to audit.md "Operational notes"
linking the flag and the manifest field.

Why "warning, not docs-only"

The docs path was suggested but rejected — the existing docs already
state the offline default clearly. Repeating the warning in CLI help
won't reach users who don't pass --help. A runtime warning fires
exactly when the user is about to be confused.

A. Smart warning (preferred)

When the rendered plan contains action-SHA placeholders AND
--resolve is off, emit a one-line warning to stderr:

Notice: --resolve off; N action-SHA placeholders in the rendered
plan will report as drift if the target was rendered online.
Re-run with --resolve to compare against real SHAs (requires gh).

Don't fail; let the user decide. Pair with a doc paragraph in
docs/reference/audit.md's "Operational notes" section.

B. Documentation only

Keep current behaviour, add a clear caveat in the --help text and
docs explaining that audits against pack-rendered repos generally
need --resolve to be meaningful.

Acceptance criteria

  • When manifest["action_shas"] is non-empty and skip_resolve
    is True, run_audit emits a one-line note to stderr naming
    the count and the recommended remediation (--resolve).
  • When --resolve is passed (online), no notice is emitted.
  • When the pack declares no action_shas, no notice is emitted
    regardless of --resolve.
  • docs/reference/audit.md "Operational notes" gains a
    paragraph linking the manifest field and the flag, with a
    copy-pasteable --resolve example.
  • Test in tests/test_audit.py asserts the notice is emitted
    (capsys / caplog) only in the warning-eligible case.

Effort

~30 minutes including tests + doc paragraph.

Source

Bot review on PR #51 (Grippy audit.py:175 MEDIUM
"Network-Skipping Mode May Cause Audits to Miss SHA Drift"),
sharpened post-Codex review (the original "silent footgun" framing
was wrong — docs do cover this — so this is a UX-warning
improvement, not a bug fix).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions