UserPromptSubmit rules are difficult to write effectively (explore learning-based context injection)

[jssblck]

Current Challenge

UserPromptSubmit rules are powerful for injecting helpful context to the agent, but they're difficult to write effectively:

Too broad → False positives/noise:

# Triggers on many innocent developer prompts
pattern: "(?i)(test|try|run).*(change|local|dev)"
# Would fire on: "test the feature", "try running it", "run the tests", etc.

Too narrow → Miss relevant cases:

# Only catches exact phrasing
pattern: "(?i)testing daemon changes"
# Misses: "modify daemon", "update background service", "change upload logic"

The Core Problem

Regex patterns can't understand semantic intent. We need to know:

• What is the developer actually trying to accomplish?
• Has the agent seen this pattern before in this session?
• What mistakes has the agent made recently?
• What context would actually be helpful to inject right now?

Potential Vision: Learning-Based Context Injection

Imagine Nudge could learn from developer-agent interactions and inject context intelligently:

1. Track Interaction Patterns Locally

• Store developer prompts and subsequent agent tool calls in a local database
• Observe sequences: "Developer asks X → agent does Y → developer asks about Z"
• Build a model of project-specific workflows

2. Semantic Analysis

• Use an LLM to understand developer prompt intent (not just regex matching)
• Recognize when developer is asking about familiar patterns
• Understand synonyms and variations ("daemon", "background service", "upload process")

3. Frequency-Based Injection

• Only inject reminders after seeing a pattern multiple times
• Track what reminders have been shown to the agent in this session
• Cooldown periods: "Don't show this again for 1 hour"

4. Learn Project-Specific Workflows

Example learned pattern:

When developer says: "add a dependency"
Agent often does: Edit tool calls to Cargo.toml
But agent forgets: cargo autoinherit step
So inject to agent: "Remember to run cargo autoinherit after adding dependencies"
After: First time in session OR if agent forgot in last 3 attempts

Real-World Example

From our project:

Workflow we want to remind the agent about:

When testing local changes to hurry, use hurry-dev instead of hurry. Run make install-dev after making changes.

Problem with regex approach:

# Too broad - fires constantly on innocent prompts
pattern: "(?i)(test|try|run)"

# Too narrow - misses variations
pattern: "(?i)test local changes"

With learning system:

• Nudge observes: Agent modifies packages/hurry/src/*
• Developer then says: "let's test this" or "try running it" or "does this work"
• System learns: After hurry modifications, agent often tests without hurry-dev
• Injects context to agent: Only on first test attempt after modifications in this session
• Learns over time: If agent consistently uses hurry-dev, stop reminding

Implementation Ideas

Option 1: Local Pattern Database

• SQLite database: (timestamp, developer_prompt, agent_tool_calls, file_changes)
• Periodic analysis: "What patterns emerge from last 100 sessions?"
• Generate dynamic rules based on learned patterns

Option 2: LLM-Based Intent Recognition

• Send developer prompt to LLM: "What is the developer trying to do?"
• Match against known workflow patterns
• Inject relevant context to agent based on semantic understanding
• Could use local/small models for privacy

Option 3: Hybrid Approach

• Learn common sequences with local database
• Use LLM for semantic matching of new developer prompts to learned patterns
• Frequency tracking to avoid noise

Privacy Considerations

• All data stored locally (never sent to servers)
• Developer control over what's tracked
• Opt-in for LLM-based analysis
• Clear data anytime

Questions

1. Is learning-based context injection aligned with Nudge's vision?
2. Would local LLM integration be feasible for semantic analysis?
3. Should this be a separate feature/mode or core functionality?
4. What's the right balance between helpful reminders and noise?

Related Use Cases

Other scenarios that would benefit:

• Migration workflows: Detect schema changes → remind agent about migration generation
• Testing patterns: Detect database code → suggest #[sqlx::test] attribute to agent
• Pre-commit reminders: After agent makes significant changes → suggest make precommit
• Pattern discovery: When developer asks to implement new features → remind agent to check for similar examples

All of these are difficult to express as simple regex patterns but would be valuable as intelligent, context-aware reminders.

Current Workaround

For now, we're sticking with PreToolUse blocker rules that have clear, enforceable patterns. But this leaves a lot of valuable contextual guidance unexpressed.