Skip to content

fix: restore dead severity signal, per-portal resilience, PII audit correctness#21

Merged
dkships merged 3 commits into
mainfrom
quality/scoring-resilience
Jul 2, 2026
Merged

fix: restore dead severity signal, per-portal resilience, PII audit correctness#21
dkships merged 3 commits into
mainfrom
quality/scoring-resilience

Conversation

@dkships

@dkships dkships commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Fixes from an adversarially verified quality audit. Two commits: scoring correctness, then resilience and PII handling. Every finding below was confirmed against the code (and the MCP SDK source for the concurrency one) before fixing.

Scoring correctness (commit 1)

  • The thread-depth severity term was dead in production. Thread bodies are never fetched, so thread_count was always 0 and min(0 * 10, 50) contributed nothing, while the README and methodology present thread depth as a severity input. The count now comes from the HelpScout list API's threads field, which costs no extra API calls. Note the semantics: it counts all thread types (agent replies, notes), and the curve saturates at 5 threads. Curve tuning is a follow-up.
  • A zero or missing thread count now scores as the baseline 1 (|| 1 instead of ?? 1), pinned by a regression test.
  • Unparseable created_at can no longer produce NaN priority scores, and future-dated tickets no longer pin severity to the cap.
  • PII placeholders like [EMAIL REDACTED] are stripped before n-gram detection, so "email redacted" can't show up as an emerging theme (it previously could, verified empirically).
  • Deleted the unreachable thread-body fetch path (~80 lines). No caller ever set includeThreads; raw thread bodies stay excluded by design pending a privacy review. Last present at d5f985f.

Resilience and PII (commit 2)

  • One failing ProductLift portal no longer drops every portal's data. Failures become per-portal warnings in the response, in all three data tools.
  • A total fetch failure now returns isError with the warnings instead of an empty analysis that reads as success, and failed fetches are never cached.
  • The module-global PII category log is gone. The MCP SDK dispatches tool handlers concurrently, so the old swap/restore pattern could interleave audit metadata across calls (scrubbing itself was per-field and never affected). Formatting functions now take an explicit per-request sink.
  • get_feature_requests now reports pii_scrubbing_applied / pii_categories_redacted like the other tools, and stops sending commenter names (role only). Names were the one identity field that bypassed the "all text is scrubbed" guarantee.
  • Feature requests are labeled with their actual source portal instead of the filter value.
  • A malformed PRODUCTLIFT_PORTALS no longer kills the whole server at startup; it degrades to zero portals and surfaces the config error in tool descriptions and list_sources.
  • preview_only now describes exactly what is fetched and sent. It previously claimed comment text and message bodies were sent; the analysis path fetches neither.
  • Upstream API error text is PII-scrubbed before entering warnings; themes-config load failures return an actionable error.

Behavior changes to be aware of

  1. Severity scores rise for multi-thread tickets (the term was contributing 0 before).
  2. get_feature_requests output: comment objects lose the author field; responses gain the two pii_* fields and optional warnings.
  3. All-sources-failed calls now error instead of returning empty success.

Verification

  • npm run build and npm test green after each commit (40 tests, 6 new).
  • Smoke-tested against live data: synthesize_feedback (552 signals, all scores finite, severity term live, no placeholder n-grams), generate_product_plan preview_only, get_feature_requests (correct portal labels, no author fields, PII fields present), list_sources.

🤖 Generated with Claude Code

dkships and others added 2 commits July 1, 2026 19:59
The severity formula's thread-depth term was dead in production: thread
bodies were never fetched, so thread_count was always 0 and min(0*10, 50)
contributed nothing. Use the list API's threads count field instead (no
extra API calls) and treat 0/missing as the baseline 1.

Also:
- Clamp recency so an unparseable created_at can't produce NaN scores
  and future-dated tickets can't pin severity to 100
- Strip PII-redaction placeholders before n-gram tokenization so
  'email redacted' can't surface as an emerging theme
- Delete the unreachable thread-body fetch path (includeThreads was
  never set by any caller; last present at d5f985f)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- One failing ProductLift portal no longer drops every portal's data;
  failures surface as named-portal warnings in the response
- Total fetch failure (all sources) returns isError instead of an empty
  analysis that reads as success, and is never cached
- Replace the module-global PII category log with an explicit
  per-request sink; concurrent tool calls no longer interleave audit
  metadata
- get_feature_requests reports pii_scrubbing_applied and
  pii_categories_redacted, and stops sending commenter names (role only)
- A malformed PRODUCTLIFT_PORTALS degrades to zero portals with the
  error surfaced in tool descriptions instead of killing the server
- Label feature requests with their actual source portal
- Guard themes.config.json loading with an actionable error
- Scrub upstream error text in warnings; collapse whitespace in quotes
- preview_only now describes exactly what is fetched and sent

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Transitive dep of @modelcontextprotocol/sdk; npm audit flagged the
4.12.x line high (GHSA-xrhx-7g5j-rcj5 and related). Lockfile-only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dkships dkships merged commit 39276d9 into main Jul 2, 2026
2 checks passed
@dkships dkships deleted the quality/scoring-resilience branch July 2, 2026 03:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant