feat(eval): Add gate-check structured output and --debug-json instrumentation

[oshorefueled]

Why

Previous evaluations used a reasoning field to improve accuracy, but reasoning tokens are generated after the model has already pattern-matched to a conclusion — making the reasoning cosmetic and post-hoc. This PR replaces that with a "tax" approach: the model is forced to emit specific boolean gate checks per violation candidate before it can proceed. Serializing through these gates constrains what the model can plausibly output next, producing more accurate evaluations without a second model pass.

What

• Requires line, rule_quote, confidence, reasoning, and gate check fields (rule_supports_claim, evidence_exact, context_supports_violation, plausible_non_violation, fix_is_drop_in, fix_preserves_meaning) in LLM structured output for both check and judge modes
• Adds deterministic violation-filter.ts — computes surface/hide decisions and reason strings from gate check booleans; no extra model call
• Adds --debug-json CLI flag that writes per-run artifacts to .vectorlint/runs/ containing raw model output, filter decisions, and surfaced violations
• Updates the built-in directive to reflect the raw-candidates + gate-checks approach
• Adds tests/evaluations/ with test rules and config for manual evaluation runs

Scope

In scope

• Structured output schema contract (both check and judge)
• Deterministic gate filtering logic
• --debug-json instrumentation
• Directive wording update

Behavior impact

• Violation output will change. The gate check fields are part of the structured output the model generates on every run — not just when --debug-json is passed. Because the model has to commit to boolean gate checks per violation candidate during generation, the set of violations it produces will differ from the old approach. Expect changes in what gets surfaced across all output formats (line, json, vale-json, rdjson), even though the output structure is unchanged.
• The standard output formats (--output json etc.) do not include the new gate fields — they are consumed internally and written to debug artifacts only
• Models that fail to populate the required gate fields will throw a structured output error
• --debug-json writes to .vectorlint/runs/ (gitignored); no impact on default runs
• Directive rewrite changes the system prompt on every evaluation, compounding the change in violation output

Risk

• Medium — the new required fields change the LLM output contract; any provider or model that doesn't support structured output reliably may fail where it previously succeeded
• Gate checks are computed on every run but only used when --debug-json is passed — adds token cost even when not debugging

How to test / verify

Checks run

• npm run test:run — passed
• npm run lint — passed

Manual verification

# Smoke test with debug output
npm run dev -- tests/evaluations/TEST_FILE.md \
  --config tests/evaluations/.vectorlint.ini \
  --debug-json --verbose

# Expected: normal findings + a line like:
# [vectorlint] Debug JSON written: .vectorlint/runs/<uuid>.json
# Verify the artifact contains raw_model_output.violations[*].line and gate fields
ls .vectorlint/runs/

Expected artifact output

When --debug-json is passed, a file is written to .vectorlint/runs/<model-tag>/<run_id>.json. Example (one violation, truncated):

{
  "run_id": "908b766e-b80a-4caf-a3f9-76a1a80aab8d",
  "timestamp": "2026-02-28T16:24:15.188Z",
  "file": "TEST_FILE.md",
  "model": {
    "provider": "openai",
    "name": "gpt-5.2-2025-12-11",
    "tag": "openai-gpt-5.2-2025-12-11"
  },
  "prompt": {
    "pack": "Test",
    "id": "Consistency",
    "filename": "consistency.md",
    "evaluation_type": "check"
  },
  "raw_model_output": {
    "reasoning": "Reviewed the input for terminology consistency...",
    "violations": [
      {
        "line": 43,
        "quoted_text": "AI-Powered Search",
        "context_before": "* **",
        "context_after": "**\n  Enable AI Search to help developers ask questions...",
        "description": "Potential terminology inconsistency: \"AI-Powered Search\" vs \"AI Search\" for the same feature.",
        "analysis": "The bullet heading names the feature \"AI-Powered Search\", but the description calls it \"AI Search\".",
        "suggestion": "Use one term consistently.",
        "fix": "  Enable AI-Powered Search to help developers ask questions about your product and instantly receive an answer.",
        "rule_quote": "Flag when the same concept is referred to by different terms",
        "checks": {
          "rule_supports_claim": true,
          "evidence_exact": true,
          "context_supports_violation": true,
          "plausible_non_violation": true,
          "fix_is_drop_in": true,
          "fix_preserves_meaning": true
        },
        "check_notes": {
          "rule_supports_claim": "The rule explicitly requires flagging the same concept referred to by different terms.",
          "evidence_exact": "Quoted text exactly matches line 43.",
          "context_supports_violation": "The very next line uses a different name (\"AI Search\") for the likely same feature.",
          "plausible_non_violation": "\"AI-Powered Search\" might be a category label while \"AI Search\" is the product name.",
          "fix_is_drop_in": "Direct substitution without structural changes.",
          "fix_preserves_meaning": "Keeps the intended meaning while aligning terminology."
        },
        "confidence": 0.8
      }
    ]
  },
  "filter_decisions": [
    {
      "index": 0,
      "surface": false,
      "reasons": ["plausible_non_violation=true"]
    }
  ],
  "surfaced_violations": []
}

Key things to verify:

• raw_model_output.violations[*] contains line, rule_quote, checks, check_notes, and confidence
• filter_decisions[*].reasons lists why each candidate was hidden or surfaced
• surfaced_violations contains only candidates that passed all gates

Rollback

• Revert to main — no DB/config/env changes
• Delete .vectorlint/runs/ locally (optional, gitignored)

Summary by CodeRabbit

Release Notes

• New Features

  • Added --debug-json CLI flag to emit debug artifacts for evaluation runs, showing model outputs and filtering decisions.
  • Enhanced violation reporting with structured quality checks and confidence scores.
  • Added five new evaluation rules for document quality: clarity, consistency, passive voice, readability, and wordiness.

• Documentation

  • Added comprehensive guides for the new gate-check feature and debug workflow.
  • Added evaluation guides and test rules documentation.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR introduces a gate-check structured output feature enabling deterministic violation filtering and debug artifact generation. It adds GateChecks and GateCheckNotes types to evaluation results, implements a violation-filter module that surfaces/hides violations based on gate boolean flags, exposes raw LLM outputs, and provides a --debug-json CLI flag to write per-run artifacts to .vectorlint/runs/. The directive is updated to request gate checks alongside raw findings.

Changes

Cohort / File(s)	Summary
Configuration .gitignore	Adds .vectorlint/runs/ to ignored paths for debug artifact storage.
Documentation docs/pr-description.md, tests/evaluations/README.md, tests/evaluations/TEST_FILE.md	Comprehensive documentation of gate-check feature, debug artifact structure, evaluation workflows, and test file template with MDX-style components.
CLI & Type Definitions src/cli/commands.ts, src/cli/types.ts, src/schemas/cli-schemas.ts	Adds --debug-json CLI flag, propagates debugJson option through EvaluationOptions and EvaluationContext, and extends CLI schema.
Core Debug Infrastructure src/debug/run-artifact.ts, src/debug/violation-filter.ts	Introduces DebugRunArtifact type and writeDebugRunArtifact() to persist artifacts; defines FilterDecision type and computeFilterDecision() for deterministic gate-based violation filtering with surface/hide logic and reason collection.
Orchestration & Model Output src/cli/orchestrator.ts	Adds debugJson pathway across evaluation flow; generates run artifacts with model info, raw_model_output, filter decisions, and surfaced violations; writes artifacts and debug logs when debugJson is enabled.
Schema & Evaluation Types src/prompts/schema.ts	Introduces GateChecks and GateCheckNotes types; extends JudgeLLMResult, CheckLLMResult, JudgeResult, and CheckResult to include line, description, rule_quote, checks, check_notes, confidence, and raw_model_output in violation objects.
Directive & Prompting src/prompts/directive-loader.ts	Updates DEFAULT_DIRECTIVE role, instructions, and output format to request two-output model (raw findings + gate checks) with explicit formatting, failure/pass semantics, and hard constraints.
Evaluator Output src/evaluators/base-evaluator.ts	Exposes raw_model_output in JudgeResult and CheckResult (single LLMResult or array for multi-chunk); aggregates per-chunk reasoning strings into single reasoning field in CheckResult.
Violation Aggregation src/scoring/scorer.ts	Updates calculateCheckScore and averageJudgeScores to preserve expanded violation shapes (line, description, rule_quote, checks, check_notes, confidence) and collects raw violations without field reconstruction.
Test Infrastructure tests/evaluations/.vectorlint.ini, tests/evaluations/test-rules/Test/*	Adds evaluation configuration, five test rules (Clarity, Consistency, Passive Voice, Readability, Wordiness), each with YAML metadata and check criteria.

Sequence Diagram

sequenceDiagram
    participant CLI as CLI
    participant Orch as Orchestrator
    participant Eval as Evaluator
    participant Filter as Filter
    participant Writer as Run Artifact Writer
    
    CLI->>Orch: evaluateFiles({ debugJson: true })
    Orch->>Eval: runEvaluation(prompt, model)
    Eval-->>Orch: CheckResult/JudgeResult + raw_model_output
    
    rect rgba(100, 150, 255, 0.5)
    Note over Orch,Writer: Debug Artifact Generation (when debugJson=true)
    Orch->>Filter: computeFilterDecision(violation)
    Filter-->>Orch: FilterDecision { surface, reasons[] }
    Orch->>Writer: writeDebugRunArtifact(runId, artifact)
    Writer-->>Orch: artifact file path
    Orch->>CLI: log debug message with path
    end
    
    CLI-->>CLI: Return aggregated results

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• Rename rule types from semi-objective/subjective to check/judge #49 — Overlapping modifications to src/prompts/schema.ts and src/evaluators/base-evaluator.ts for renaming evaluation types and extending result shapes.
• feat(cli): Refactor help output #47 — Directly conflicts with debugJson flag implementation in CLI options and orchestrator; PR #47 removes the same flag this PR adds.
• Always include user instruction in every run #56 — Modifies src/prompts/directive-loader.ts with competing directive content and instruction block changes.

Suggested reviewers

• ayo6706

Poem

🐰 Gate checks and filters, oh what a sight!
Raw outputs dancing in artifact light,
Debug flags waving through orchestration's flow,
Violations surfaced with reasons to know,
A structured tale of clarity and might! ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 41.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically summarizes the main change: introducing gate-check structured output and the --debug-json instrumentation flag, which are the central features of this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feat/eval-gate-checks

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

src/evaluators/base-evaluator.ts (1)

150-177: ⚠️ Potential issue | 🟠 Major

Capture raw_model_output only when debug collection is enabled.

Line [150]-Line [177] and Line [202]-Line [239] accumulate and return full chunk-level LLM payloads on every run. That adds avoidable memory/serialization overhead in non-debug execution. Please gate raw capture/attachment behind a debug option propagated from CLI/orchestrator.

Also applies to: 202-239

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/evaluators/base-evaluator.ts` around lines 150 - 177, The code
unconditionally collects and returns chunk-level LLM payloads (rawChunkOutputs
and raw_model_output) which should be gated by a debug flag; add a debug/config
boolean (e.g., this.collectDebug or this.options.debug) and modify the loop
around llmProvider.runPromptStructured so you only push llmResult into
rawChunkOutputs when that flag is true, and change the returned object to
include raw_model_output only when the flag is enabled; apply the same gating to
the similar block referenced (lines ~202-239) so both capture and attachment of
raw_model_output are conditional.

src/prompts/schema.ts (1)

50-50: ⚠️ Potential issue | 🟡 Minor

Harden numeric constraints for line and confidence.

Line [50] and Line [142] currently accept any number for line, and Line [99]/Line [191] accept unbounded confidence. Use integer/minimum constraints for line numbers and a [0, 1] range for confidence.

Proposed schema hardening

- line: { type: "number" },
+ line: { type: "integer", minimum: 1 },

- confidence: { type: "number" },
+ confidence: { type: "number", minimum: 0, maximum: 1 },

Also applies to: 99-99, 142-142, 191-191

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/prompts/schema.ts` at line 50, The schema currently allows unbounded
numbers for the properties named "line" and "confidence"; update each schema
definition that declares line: { type: "number" } to use an integer with a
sensible minimum (e.g., line: { type: "integer", minimum: 1 }) and update each
confidence declaration to enforce a [0,1] bound (e.g., confidence: { type:
"number", minimum: 0, maximum: 1 }); apply these changes to every occurrence of
the "line" and "confidence" properties found in this file so the validators (the
schema objects that define "line" and "confidence") enforce integer line numbers
and bounded confidence values.

🧹 Nitpick comments (6)

src/prompts/directive-loader.ts (1)

35-40: Minor duplication in hard constraints.

Lines 36-37 repeat constraints already stated in lines 20-21 ("Do NOT invent evidence" and "Every quoted span must be copied exactly"). While repetition in LLM prompts can reinforce important rules, consider whether both occurrences are intentional for emphasis or could be consolidated.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/prompts/directive-loader.ts` around lines 35 - 40, The "Hard constraints"
block contains duplicate rules ("Do NOT invent evidence." and "Every quoted span
must be copied exactly...")—edit the prompt template in directive-loader.ts to
remove the repeated lines under the "Hard constraints" section so each
constraint appears only once (keep the first occurrence for emphasis), update
the surrounding string/template accordingly, and verify any code that references
that template (e.g., the directive or prompt-building variable containing the
"Hard constraints" text) still produces the expected prompt output.

src/cli/orchestrator.ts (2)

757-792: Same recommendation: wrap debug artifact writing in try/catch for Judge path.

Applies the same defensive error handling suggestion as the Check path to prevent evaluation failures from debug artifact I/O issues.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/cli/orchestrator.ts` around lines 757 - 792, The debug JSON write in the
Judge path (guarded by debugJson) can throw and should be wrapped in a
try/catch: surround the block that computes runId, flat (from result.criteria
and computeFilterDecision), model (getModelInfoFromEnv) and the call to
writeDebugRunArtifact so any I/O/error is caught; on error call
processLogger.warn or console.warn with a clear message including the error
details and continue without failing the evaluation. Ensure the catch only
handles the debug-artifact logic (do not swallow other errors) and preserve the
existing console.warn(`[vectorlint] Debug JSON written: ${filePath}`) on
success.

---

670-695: Consider handling potential write failures for debug artifacts.

writeDebugRunArtifact performs synchronous file I/O (mkdirSync, writeFileSync) that can throw exceptions (e.g., permission denied, disk full). Since this is in the main evaluation flow, an unhandled exception would crash the evaluation. Consider wrapping in a try/catch to log a warning and continue evaluation gracefully.

🛡️ Proposed defensive wrapper

     if (debugJson) {
+      try {
         const runId = randomUUID();
         const decisions = result.violations.map((v) => computeFilterDecision(v));
         // ... rest of artifact logic ...
         console.warn(`[vectorlint] Debug JSON written: ${filePath}`);
+      } catch (e: unknown) {
+        console.warn(`[vectorlint] Failed to write debug artifact: ${e instanceof Error ? e.message : String(e)}`);
+      }
     }

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/cli/orchestrator.ts` around lines 670 - 695, The debug artifact write
block guarded by debugJson should be made exception-safe: wrap the call sequence
that generates runId and calls writeDebugRunArtifact (including uses of
randomUUID(), decisions/surfaced computation, and console.warn) in a try/catch
so that any synchronous file I/O errors do not crash evaluation; on catch, log a
non-fatal warning including the caught error (e.g., via console.warn or
processLogger) and continue without rethrowing so the rest of the evaluation
proceeds normally.

src/debug/violation-filter.ts (1)

28-43: Missing fix_empty in reasons when fix is empty.

The fixEmpty variable is computed (line 30) and used in the surface decision (line 38), but no corresponding reason is added when fix is empty. This asymmetry makes it harder to understand why a violation was not surfaced.

♻️ Proposed fix to add fix_empty reason

   if (typeof v.confidence !== "number" || v.confidence < 0.75) reasons.push("confidence<0.75");
 
   const fixEmpty = (v.fix ?? "").trim() === "";
+  if (fixEmpty) reasons.push("fix_empty");
 
   const surface =

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/debug/violation-filter.ts` around lines 28 - 43, The code computes
fixEmpty but never records it in reasons, so add a symmetric reason when the fix
is empty: after computing const fixEmpty = (v.fix ?? "").trim() === ""; check if
(fixEmpty) and push a descriptive token (e.g., "fix_empty") into the reasons
array; ensure you reference the existing variables used here (fixEmpty, reasons,
v) so the new push occurs before the surface decision that relies on fixEmpty.

src/debug/run-artifact.ts (1)

34-62: Consider documenting or validating runId format for filename safety.

The runId is used directly in the filename (${runId}.json). While UUID format from randomUUID() is safe, if this function is ever called with user-provided IDs, it could lead to path traversal or invalid filenames. Consider either documenting the contract that runId must be a safe filename segment or applying sanitizePathSegment to it as well.

🛡️ Optional defensive sanitization

-  const filePath = path.join(dir, `${runId}.json`);
+  const safeRunId = sanitizePathSegment(runId);
+  const filePath = path.join(dir, `${safeRunId}.json`);

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/debug/run-artifact.ts` around lines 34 - 62, The filename uses runId
directly which can allow unsafe segments; in writeDebugRunArtifact either
validate runId against a safe pattern (e.g., UUID or
alphanumeric/dash/underscore) or apply sanitizePathSegment before using it in
the filename and any path joins (e.g., when building filePath and any other path
operations); update the code to use the sanitizedRunId (or assert the contract)
so path traversal/invalid filename risks are eliminated while keeping existing
sanitizePathSegment for artifact.subdir.

src/prompts/schema.ts (1)

59-98: Extract shared gate schemas to avoid drift.

Line [59]-Line [98] and Line [151]-Line [190] duplicate the same checks and check_notes schema blocks. Pull these into shared constants and reuse in both builders so gate contract updates happen in one place.

Also applies to: 151-190

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/prompts/schema.ts` around lines 59 - 98, The two identical schema objects
named checks and check_notes should be extracted into shared constants (e.g.,
gateChecksSchema and gateCheckNotesSchema) and reused in both builders instead
of duplicating the inline objects; locate the inline definitions for checks and
check_notes (the objects with additionalProperties:false, properties
{...boolean/string...} and the required arrays) and replace them with references
to the new constants so the properties, additionalProperties and required arrays
are preserved and future changes only need to be made in one place.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/pr-description.md`:
- Around line 55-63: The debug artifact path examples are inconsistent: replace
the shorter `.vectorlint/runs/<uuid>.json` example with the detailed
`.vectorlint/runs/<model-tag>/<run_id>.json` form so all references match;
update the header text, the `ls .vectorlint/runs/` example (or guidance) and the
subsequent "When `--debug-json` is passed..." sentence to consistently use
`.vectorlint/runs/<model-tag>/<run_id>.json` (ensure any mentions of `<uuid>`
are changed to `<model-tag>/<run_id>` and keep the rest of the example text
intact).

In `@tests/evaluations/.vectorlint.ini`:
- Line 1: Update the RulesPath in the .vectorlint.ini so it points to the test
rule pack directory (set RulesPath = ./test-rules) to ensure the pack named by
RunRules = Test (located under ./test-rules/Test) is discoverable; modify the
RulesPath entry (the RulesPath setting in .vectorlint.ini) accordingly so the
custom rules are loaded during manual evaluations.

---

Outside diff comments:
In `@src/evaluators/base-evaluator.ts`:
- Around line 150-177: The code unconditionally collects and returns chunk-level
LLM payloads (rawChunkOutputs and raw_model_output) which should be gated by a
debug flag; add a debug/config boolean (e.g., this.collectDebug or
this.options.debug) and modify the loop around llmProvider.runPromptStructured
so you only push llmResult into rawChunkOutputs when that flag is true, and
change the returned object to include raw_model_output only when the flag is
enabled; apply the same gating to the similar block referenced (lines ~202-239)
so both capture and attachment of raw_model_output are conditional.

In `@src/prompts/schema.ts`:
- Line 50: The schema currently allows unbounded numbers for the properties
named "line" and "confidence"; update each schema definition that declares line:
{ type: "number" } to use an integer with a sensible minimum (e.g., line: {
type: "integer", minimum: 1 }) and update each confidence declaration to enforce
a [0,1] bound (e.g., confidence: { type: "number", minimum: 0, maximum: 1 });
apply these changes to every occurrence of the "line" and "confidence"
properties found in this file so the validators (the schema objects that define
"line" and "confidence") enforce integer line numbers and bounded confidence
values.

---

Nitpick comments:
In `@src/cli/orchestrator.ts`:
- Around line 757-792: The debug JSON write in the Judge path (guarded by
debugJson) can throw and should be wrapped in a try/catch: surround the block
that computes runId, flat (from result.criteria and computeFilterDecision),
model (getModelInfoFromEnv) and the call to writeDebugRunArtifact so any
I/O/error is caught; on error call processLogger.warn or console.warn with a
clear message including the error details and continue without failing the
evaluation. Ensure the catch only handles the debug-artifact logic (do not
swallow other errors) and preserve the existing console.warn(`[vectorlint] Debug
JSON written: ${filePath}`) on success.
- Around line 670-695: The debug artifact write block guarded by debugJson
should be made exception-safe: wrap the call sequence that generates runId and
calls writeDebugRunArtifact (including uses of randomUUID(), decisions/surfaced
computation, and console.warn) in a try/catch so that any synchronous file I/O
errors do not crash evaluation; on catch, log a non-fatal warning including the
caught error (e.g., via console.warn or processLogger) and continue without
rethrowing so the rest of the evaluation proceeds normally.

In `@src/debug/run-artifact.ts`:
- Around line 34-62: The filename uses runId directly which can allow unsafe
segments; in writeDebugRunArtifact either validate runId against a safe pattern
(e.g., UUID or alphanumeric/dash/underscore) or apply sanitizePathSegment before
using it in the filename and any path joins (e.g., when building filePath and
any other path operations); update the code to use the sanitizedRunId (or assert
the contract) so path traversal/invalid filename risks are eliminated while
keeping existing sanitizePathSegment for artifact.subdir.

In `@src/debug/violation-filter.ts`:
- Around line 28-43: The code computes fixEmpty but never records it in reasons,
so add a symmetric reason when the fix is empty: after computing const fixEmpty
= (v.fix ?? "").trim() === ""; check if (fixEmpty) and push a descriptive token
(e.g., "fix_empty") into the reasons array; ensure you reference the existing
variables used here (fixEmpty, reasons, v) so the new push occurs before the
surface decision that relies on fixEmpty.

In `@src/prompts/directive-loader.ts`:
- Around line 35-40: The "Hard constraints" block contains duplicate rules ("Do
NOT invent evidence." and "Every quoted span must be copied exactly...")—edit
the prompt template in directive-loader.ts to remove the repeated lines under
the "Hard constraints" section so each constraint appears only once (keep the
first occurrence for emphasis), update the surrounding string/template
accordingly, and verify any code that references that template (e.g., the
directive or prompt-building variable containing the "Hard constraints" text)
still produces the expected prompt output.

In `@src/prompts/schema.ts`:
- Around line 59-98: The two identical schema objects named checks and
check_notes should be extracted into shared constants (e.g., gateChecksSchema
and gateCheckNotesSchema) and reused in both builders instead of duplicating the
inline objects; locate the inline definitions for checks and check_notes (the
objects with additionalProperties:false, properties {...boolean/string...} and
the required arrays) and replace them with references to the new constants so
the properties, additionalProperties and required arrays are preserved and
future changes only need to be made in one place.

---

ℹ️ Review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f4c55ab and c8af0d9.

📒 Files selected for processing (20)

• .gitignore
• docs/pr-description.md
• src/cli/commands.ts
• src/cli/orchestrator.ts
• src/cli/types.ts
• src/debug/run-artifact.ts
• src/debug/violation-filter.ts
• src/evaluators/base-evaluator.ts
• src/prompts/directive-loader.ts
• src/prompts/schema.ts
• src/schemas/cli-schemas.ts
• src/scoring/scorer.ts
• tests/evaluations/.vectorlint.ini
• tests/evaluations/README.md
• tests/evaluations/TEST_FILE.md
• tests/evaluations/test-rules/Test/clarity.md
• tests/evaluations/test-rules/Test/consistency.md
• tests/evaluations/test-rules/Test/passive-voice.md
• tests/evaluations/test-rules/Test/readability.md
• tests/evaluations/test-rules/Test/wordiness.md

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Align the debug artifact path examples.

Line [55] shows .vectorlint/runs/<uuid>.json, while Line [62] shows .vectorlint/runs/<model-tag>/<run_id>.json. Please make these consistent so verification targets the correct location.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@docs/pr-description.md` around lines 55 - 63, The debug artifact path
examples are inconsistent: replace the shorter `.vectorlint/runs/<uuid>.json`
example with the detailed `.vectorlint/runs/<model-tag>/<run_id>.json` form so
all references match; update the header text, the `ls .vectorlint/runs/` example
(or guidance) and the subsequent "When `--debug-json` is passed..." sentence to
consistently use `.vectorlint/runs/<model-tag>/<run_id>.json` (ensure any
mentions of `<uuid>` are changed to `<model-tag>/<run_id>` and keep the rest of
the example text intact).

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Fix RulesPath to target the added test rule pack.

Line 1 points to ./builtin, but this PR’s test rules are under ./test-rules/Test. With RunRules = Test, this can prevent the intended rules from being loaded in manual evaluations.

🔧 Proposed fix

-RulesPath = ./builtin
+RulesPath = ./test-rules

Based on learnings: Organize custom rules into subdirectories (packs) within RulesPath.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/evaluations/.vectorlint.ini` at line 1, Update the RulesPath in the
.vectorlint.ini so it points to the test rule pack directory (set RulesPath =
./test-rules) to ensure the pack named by RunRules = Test (located under
./test-rules/Test) is discoverable; modify the RulesPath entry (the RulesPath
setting in .vectorlint.ini) accordingly so the custom rules are loaded during
manual evaluations.