Skip to content

Fix(chat): remove internal MCP tool name from VendorChatAssistant RULES#473

Open
Jean-Regis-M wants to merge 1 commit intoGenAI-Security-Project:mainfrom
Jean-Regis-M:patch-56
Open

Fix(chat): remove internal MCP tool name from VendorChatAssistant RULES#473
Jean-Regis-M wants to merge 1 commit intoGenAI-Security-Project:mainfrom
Jean-Regis-M:patch-56

Conversation

@Jean-Regis-M
Copy link
Copy Markdown
Contributor

@Jean-Regis-M Jean-Regis-M commented Apr 10, 2026

Summary

Fixes #444 Removes internal MCP tool name findrive__get_file from RULES section in VendorChatAssistant._get_system_prompt(), establishing consistent plain-language abstraction across CAPABILITIES and RULES.

Problem

The Vendor prompt has inconsistent abstraction levels for the same capability:

  • Line 500 (CAPABILITIES): Describes FinDrive in plain language "Browse, search, and read files stored in FinDrive"
  • Line 517 (RULES): Explicitly names internal MCP implementation "read them using the findrive__get_file tool"

This creates ambiguity for developers maintaining the prompt (no clear convention for when tool names appear) and violates test_chat_prompt_055 which prohibits internal tool name leakage.

Root Cause

The prompt evolved organically without a governing convention. CAPABILITIES was written for user comprehension (plain language), while RULES was written for tool dispatch instruction (implementation detail). CoPilotAssistant already follows the correct plain-language-only pattern.

Solution

Single-line change Replace explicit findrive__get_file with generic "the appropriate file reading tool" in line 517.

This:

  • Removes internal implementation detail from user-facing prompt
  • Matches CoPilotAssistant's plain-language convention
  • Preserves semantic meaning (LLM still knows to read files via available MCP tools)
  • Passes test_chat_prompt_055 (no internal name leakage)

Impact

Category Status
Breaking changes None
Diff size 1 line
Regression risk Zero
Backward compatibility Preserved
Test failures None (existing leak test now passes)
Developer clarity Consistent convention established

Testing

  1. Leakage check: "findrive__get_file" not in prompt → PASS
  2. Semantic preservation: "appropriate file reading tool" in prompt → PASS
  3. Regression suite: pytest tests/test_chat_prompt_055.py -v → PASS
  4. Full chat tests: pytest tests/test_chat.py → PASS (no behavioral change)

Tasks Done

  • Identified inconsistency between CAPABILITIES (line 500) and RULES (line 517)
  • Confirmed CoPilotAssistant uses correct plain-language pattern as reference
  • Removed explicit findrive__get_file from line 517
  • Replaced with generic "the appropriate file reading tool"
  • Verified no other internal tool names appear in Vendor prompt
  • Confirmed test_chat_prompt_055 passes after change
  • Validated LLM behavior unchanged (MCP dispatch still works via _mcp_provider)
  • Ensured no other sections reference internal MCP tool names

PR Verification Checklist

  • Reviewer can understand in <10 seconds
  • Diff is mathematically minimal (1 line, 1 file)
  • Every changed line is necessary
  • Causality is undeniable (plain language ↔ plain language)
  • No assumptions made without evidence
  • All CI checks will pass without iteration

Root cause:
CAPABILITIES describes FinDrive in plain language but RULES explicitly
names findrive__get_file, creating inconsistent abstraction levels.

Solution:
Replace explicit tool name with generic "appropriate file reading tool"
to match plain-language convention used in CAPABILITIES.

Impact:
- No behavioral change (LLM still uses MCP dispatch internally)
- No breaking changes
- Aligns with CoPilotAssistant plain-language pattern
- test_chat_prompt_055 (no internal name leakage) now passes

Signed-off-by: JEAN REGIS <240509606@firat.edu.tr>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant