feat(oath-keeper): turn_end delivery, configurable 4-stage pipeline, classifier model, TUI dashboard#66
Open
coda-rho-bot wants to merge 51 commits into
Open
Conversation
… StateStore Major rewrite since competition submission: - LLM-based promise detection (replaces regex pre-filter) - Queued delivery state (pending → queued → delivering → delivered) - StateStore builder pattern with enforced save() - Verbose logging via flag file (touch ~/.letta/mods/oath-keeper.verbose) - Silent by default — no CLI spam - turn_end + polling hybrid detection with dedup - False positive tracking - Debug log to disk (capped at 500 entries)
1. Redirect stderr on entire execSync pipeline (grep -P can write to stderr on macOS, leaking console output even when logging is off). Also use stdio: pipe to suppress any remaining stderr. 2. turn_end handler now uses event.assistantMessage directly instead of calling fetchLatestAgentMessage() (API round-trip). Dedup via promise text similarity instead of message ID. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
The turn_end handler now extracts conversationId and agentId from the event/ctx parameters instead of the static oath-env.json file. This means oaths are delivered back to the conversation that originated them, not whatever conversation the env file points at. The env file remains as fallback for the polling path (listener mode has no event context, only one conversation). Also passes oath.agentId to isConversationBusy() so the busy check queries the correct agent's messages. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
The LLM now extracts a delay_seconds field when classifying a promise.
If the agent specified a time ("in 5 minutes" → 300s), the LLM uses
that. If no time was specified, the LLM estimates based on task
complexity (quick check → 60-120s, investigation → 300-600s, deep
work → 900+s). Clamped to 30s minimum, 86400s (24h) maximum.
Default fallback of 5 minutes if the LLM omits the field or gives an
invalid value.
Replaces the hardcoded 60-second constant for all oaths.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta Code <noreply@letta.com>
Root cause of duplicate oaths: when turn_end is available, both turn_end AND polling were scanning for promises. The polling path used the env file conversation and created duplicates with different promise text/delay values. Split pollCycle into: - pollDeliveryCycle: handles delivery timing, queue transitions, stuck recovery, and conversation history checks. No scanning. Runs on every poll regardless of mode. - pollCycle: calls pollDeliveryCycle + scans for new promises. Only used when turn_end is NOT available (listener/desktop). When turn_end IS available: only pollDeliveryCycle runs (every 15s for delivery timing). Detection is exclusively via turn_end events. This eliminates the duplicate detection path entirely. Also removes delay clamps (min/max) per user request — LLM has full control over delay_seconds. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
- Polling no longer scans for promises when turn_end handles detection. Eliminates duplicate oaths with inconsistent delays. - pollDeliveryCycle handles delivery timing/checks in both modes. - pollCycle runs delivery + scanning (listener mode only). - Removed all delay clamps — LLM determines delay freely. - turnEventsActive flag gates scanning section. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
Standalone Rust binary that reads the mod's state JSON and renders pending/delivering/delivered oaths in a live terminal UI. Build: cd packages/oath-keeper/cli && cargo build --release Run: ./target/release/oath-keeper 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
- turn_end detection (primary) + polling fallback - LLM-based detection with delay extraction - Conversation scoping via event context - TUI dashboard build/run instructions - Verbose flag documentation - Updated architecture and lifecycle details 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
When a new promise is detected, the LLM compares it against all active oaths to check for semantic duplicates. This catches paraphrased duplicates that string matching misses (e.g., "tell you the time" vs "Tell the user the time in 20 seconds"). String-based dedup remains as a fast fallback. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
Per Cameron (Letta team): the REST API bypasses the runtime, so tools
don't work during delivery turns. Using turn_end with { continue: "..." }
injects the delivery prompt through the normal turn pipeline — tools,
queuing, and channel context all work properly.
Architecture change:
- turn_end handler now does triple duty: delivery (via { continue }),
delivered-marking, and promise detection
- pollDeliveryCycle queues due oaths but does NOT deliver via REST
when turn_end is active
- REST API delivery (tryDeliverOath) remains as fallback for listener
mode where turn_end is not available
- Delivery timing: oath queued by timer → delivered on next turn_end
via { continue }
This should fix the doom loop where tool calls (web_search) fired
uncontrollably during delivery turns.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta Code <noreply@letta.com>
Tools now work during delivery turns via turn_end { continue }, so:
- Removed "Do NOT use any tools" constraint
- Removed "1-3 sentences" output length restriction
- Removed stale comments about tool limitations
- Explicitly grants tool access
- Includes user context when available (not "(turn_end)")
- Keeps [Oath Delivered] marker for mod detection
The agent is smart enough to determine how to fulfill its promise —
don't over-instruct.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta Code <noreply@letta.com>
False positives are now tracked with their own status ("false_positive")
instead of being mixed into "failed". This makes it clear in both the
list_oaths tool and the TUI which oaths were genuine failures vs LLM
rejections.
Changes:
- New status: "false_positive" in Oath interface
- logFalsePositive uses "false_positive" instead of "failed"
- list_oaths tool shows false positives as FP: entries
- TUI header shows FP: count in dark gray
- TUI list and detail views show FALSE POS badge in dark gray
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta Code <noreply@letta.com>
Promise classification and dedup now use a separate configurable agent
instead of always using the main agent. This allows users to point
classification at a cheaper/faster model.
Config file: ~/.letta/mods/oath-keeper.config.json
{
"classifierAgentId": "agent-xxx"
}
Defaults to the main agent if not set.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta Code <noreply@letta.com>
Replace the no-op detectPromiseRegex with a weighted n-gram scoring system. Messages are only sent to the LLM classifier if they contain promise-like language patterns. Pre-filter stages: - Skip short messages (<15 chars) - Skip code-heavy messages (>5% syntax characters) - Score against weighted promise n-grams (strong: 3.0, moderate: 2.0-2.5, weak: 1.0-1.5) - Send to LLM only if score >= 2.0 Expected to eliminate 70-80% of messages before LLM classification, with near-zero false negative rate for genuine promises. Academic backing: Cascaded Signal Refinement paper showed lexical pre-filtering reduced 88.6% of messages before LLM screening. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
Co-Authored-By: Letta Code <noreply@letta.com>
README now covers:
- Three-stage detection (n-gram pre-filter → LLM classification → LLM dedup)
- turn_end { continue } delivery mechanism
- Configurable classifier agent
- Oath lifecycle including false_positive status
- TUI with false positive display
- Listener/desktop fallback configuration
- Verbose logging flag
N-gram pre-filter threshold set to >1.5.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta Code <noreply@letta.com>
Mod changes: - Messages rejected by n-gram pre-filter now logged with status "prefilter_rejected" instead of silently dropped - Oath entries store ngramScore when pre-filter passes - list_oaths tool shows prefiltered entries and scores TUI changes: - New PREFILTER badge (magenta) for prefilter_rejected status - Header shows PF: count - List view shows [score] next to promise when available - Detail view shows Score: field This enables full debugging of the detection pipeline — every message is now visible in the TUI with its status and score. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
… rejections - false_positive entries now include the ngram score that triggered the LLM check - prefilter_rejected entries include the computed score (even though it was below threshold) for debugging - computeNgramScore() extracted as standalone for reuse - list_oaths and TUI both show scores for all entry types 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
- Removed Mode::FalsePositives and 'f' key (redundant — false positives are now visible in the main list with their own status badge) - Added 'c' key: clear all filtered entries (prefilter_rejected + false_positive) - Added 'C' key: clear all completed entries (delivered, failed, false_positive, prefilter_rejected) — keeps only active oaths 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
TUI: - n-gram score moved from line 1 (after promise) to line 2 with done/src/age/conv as "score: N" - Source label now uses deliveryMode from the oath (turn_end/polling) instead of hardcoded "polled from letta mod" - Conversation ID shows "N/A" when empty instead of blank Mod: - logFalsePositive and logPreFilterRejection now accept and store conversationId and agentId from the calling context - turn_end handler passes eventConvId/eventAgentId to both functions 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
Co-Authored-By: Letta Code <noreply@letta.com>
Previous edit was lost during file sync. Redone properly. Co-Authored-By: Letta Code <noreply@letta.com>
…scores - Status badge table with colors and descriptions - Keyboard controls reference - TUI launch flags (--plain, --purge) - Updated list_oaths description with new statuses - State stores n-gram scores for debugging 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
Config file (~/.letta/mods/oath-keeper.config.json) now supports: - ngramFilter: enable/disable n-gram pre-filter (default: true) - llmConfirm: enable/disable LLM promise classification (default: true) - llmDedup: enable/disable LLM semantic dedup (default: true) Safety: if both ngramFilter and llmConfirm are disabled, no oaths are created — the mod skips detection entirely. When n-gram is disabled, all messages skip the pre-filter and go directly to LLM confirmation (or oath creation if LLM is also off). When LLM confirm is disabled, messages that pass n-gram scoring create oaths directly without LLM classification. TUI displays filter status in header: - NGRAM:on/off (green/red) - LLM:on/off (green/red) - DEDUP:on/off (green/dark gray) - Warning in red bold if all filters are off Mod writes filter status to ~/.letta/mods/oath-keeper-filter-status.json on activation for the TUI to read. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
The mod now fetches the classifier agent's model at activation time and writes it to the filter status file. The TUI displays it in the header: "Model: lc-zai-coding/glm-5.1" To configure which model handles classification, set classifierAgentId in the config file to an agent that uses the desired model. Co-Authored-By: Letta Code <noreply@letta.com>
- Configuration section with full config JSON example - Filter toggle table with defaults and descriptions - Safety note about both filters being disabled - TUI header display section with filter status and model - Updated oath lifecycle with prefilter_rejected status - Updated architecture to mention configurable filters Co-Authored-By: Letta Code <noreply@letta.com>
- Rust struct fields now use #[serde(rename)] to match JSON camelCase (llmConfirm, llmDedup, filtersActive, classifierAgentId, classifierModel) - Header chunk increased to Length(3) to fit filter status on second line without overlapping the oath list Co-Authored-By: Letta Code <noreply@letta.com>
Previously both were rendered into the same chunk as separate widgets, causing the second to overwrite the first. Now combined into one Paragraph with two Lines. Co-Authored-By: Letta Code <noreply@letta.com>
New config option: negativeFilter (default: true) - Skips messages < 15 characters - Skips code-heavy messages (> 5% syntax characters) - When disabled, all messages proceed to n-gram scoring regardless TUI header now shows: NEG:on/off NGRAM:on/off LLM:on/off DEDUP:on/off Co-Authored-By: Letta Code <noreply@letta.com>
- Line 1: status badge + promise text - Line 2: done/timer + src + age + score - Line 3: conv + agent IDs Co-Authored-By: Letta Code <noreply@letta.com>
Line 3 now shows: conv:Oath-Keeper (conv-289a586a) agent:Coda (agent-b499137a) Fetches names from the Letta API via curl. Falls back to 'unknown' if API is unreachable. Co-Authored-By: Letta Code <noreply@letta.com>
get_env() now verifies the env file's port is alive (curl health check). If dead, falls back to ss port discovery (same as the mod). This ensures the TUI resolves agent/conv names correctly after app restarts. Co-Authored-By: Letta Code <noreply@letta.com>
- New config option: ngramThreshold (default: 1.25) - TUI filter line shows threshold: NGRAM:on(>1.25) - detectPromiseRegex uses configurable threshold instead of hardcoded 1.5 Co-Authored-By: Letta Code <noreply@letta.com>
The env file's port goes stale on every app restart. process.env is always current in the mod runtime. Priority is now: 1. process.env.LETTA_BASE_URL (current) 2. env file (fallback for listener mode) 3. ss port discovery (final fallback) 4. localhost:8283 (last resort) Fixes 'Model: unknown' in TUI — the model fetch was hitting a dead port. Co-Authored-By: Letta Code <noreply@letta.com>
Classification and dedup LLM calls now create throwaway conversations
with a specific model. Default is letta/auto-fast (cheap, fast).
New config option: classifierModel (default: "letta/auto-fast")
The model is passed when creating the throwaway conversation via
POST /v1/conversations with {"model": "letta/auto-fast"}.
This means users get cheap classification out of the box without
needing to set up a separate classifier agent.
Co-Authored-By: Letta Code <noreply@letta.com>
Co-Authored-By: Letta Code <noreply@letta.com>
Performance:
- Agent and conversation name lookups are now cached in a static
HashMap. Previously every render frame made 2 curl calls per entry,
causing severe lag with many entries.
UX:
- Loading screen on launch ("Loading Oath Keeper...")
- Shutdown screen on quit ("Shutting down...")
Co-Authored-By: Letta Code <noreply@letta.com>
- Four-stage detection (added Stage 0 negative filter) - Configurable ngramThreshold (default: 1) in config table and JSON - Configurable classifierModel (default: letta/auto-fast) - Updated all threshold references from 1.5 to 1 - TUI header shows NGRAM with threshold - TUI entries show resolved conv + agent names - Classifier model section explaining per-conversation model override Co-Authored-By: Letta Code <noreply@letta.com>
- prefilter_rejected entries pruned after 1 hour (was never pruned) - false_positive entries pruned after 6 hours (was 24h bucket) - delivered/failed still pruned after 24 hours These statuses are debug/audit entries — they don't need to accumulate indefinitely and bloat the state file (60+ entries was causing TUI lag). Co-Authored-By: Letta Code <noreply@letta.com>
getApiConfig() now verifies the port is alive (curl health check) every 60 seconds. If the port is dead (app restarted), it automatically falls back to ss discovery to find the new port. This means the mod survives app restarts without manual /reload — the polling interval will self-correct to the new port within 60 seconds. Co-Authored-By: Letta Code <noreply@letta.com>
Co-Authored-By: Letta Code <noreply@letta.com>
The activate function was marked async but has no top-level await calls. The Letta Code mod framework likely doesn't handle async activate functions properly, silently failing to load the mod after app restarts. All await calls are inside nested event handler callbacks, not in the activate body itself. Co-Authored-By: Letta Code <noreply@letta.com>
Agent: coda-rho-bot Email: coda-github@mail.angussoftware.com 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
turn_end { continue } delivery requires an active conversation — if the
user doesn't send another message after the oath becomes due, the oath
sits in queued forever.
Fix: if an oath has been queued for more than 60 seconds without a
turn_end firing, fall back to REST API delivery.
Co-Authored-By: Letta Code <noreply@letta.com>
- deliveryMode now updates on delivery: "turn_end" or "rest_api" (was set at creation time, misleading about how oath was delivered) - "Confirmed via conversation history" replaced with actual response text - TUI src field shows "unknown" instead of "mod" when no deliveryMode Co-Authored-By: Letta Code <noreply@letta.com>
Author
|
@carenthomas — would appreciate a review when you get a chance. This is the continued development of Oath Keeper (previously PR #57). Major additions since last review:
Thanks! |
Zero LLM calls by default. Oaths are created directly from messages that pass the n-gram pre-filter. Users opt in to LLM features: - llmConfirm: enables LLM promise classification (fewer false positives) - llmDedup: enables LLM semantic dedup (catches paraphrased duplicates) When both are off (default): - N-gram filter runs (if enabled) - Messages below threshold → prefilter_rejected, no LLM call - Messages above threshold → oath created directly, no LLM call - Only string-based dedup runs (no LLM) Verified code flow: confirmPromise() and isDuplicatePromise() are never called when their respective toggles are off. Co-Authored-By: Letta Code <noreply@letta.com>
The mod now captures token usage from the usage_statistics SSE event during confirmPromise and isDuplicatePromise calls. Token counts (prompt, completion, total) are stored on each oath entry. TUI display: - List view: "tokens:1234" on line 2 (after score) - Info view: "Tokens: 1234 (prompt: 1100, completion: 134)" Entries with no LLM calls (n-gram only or LLM disabled) show "tokens:0" or "0 (no LLM calls)" respectively. Co-Authored-By: Letta Code <noreply@letta.com>
…ess spawns every frame TUI was spawning `letta cron list` subprocess and curl health-check on every 500ms refresh cycle. Now caches: crons (30s TTL), env (60s TTL), filter status (5s TTL). Poll interval increased to 1s. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
The env file (oath-env.json) had a hardcoded port that became stale on every app restart (port changes each time). The mod had ss discovery fallback, but the stale env file caused confusion and delays. Now on activation, the mod: 1. Checks if the env file port is alive 2. If dead, discovers the current port via ss 3. Updates the env file with the correct port 4. Caches the result for 60s Also: removed redundant double health-check curl in getApiConfig, deduplicated ss discovery into a shared discoverPort() function. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
Keys: 1=NEG 2=NGRAM 3=LLM 4=DEDUP 5=cycle model Writes to oath-keeper.config.json and updates filter-status.json for immediate display feedback. Mod picks up changes on next /reload. 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
Replaces hardcoded model list with fetch_models() that calls GET /v1/models and caches the result. Filters out lc-openrouter/ (massive Letta Cloud OpenRouter proxy catalog). Falls back to letta/auto-fast + letta/auto if API is unreachable. Status bar shows position in cycle: "Model: zai/glm-5.2 (5/42)" 👾 Generated with [Letta Code](https://letta.com) Co-Authored-By: Letta Code <noreply@letta.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Major rewrite of Oath Keeper since the competition submission (PRs #28, #33, #35).
Delivery
Four-stage detection pipeline
negativeFiltertruengramFiltertruellmConfirmtruellmDeduptrueAll stages individually toggleable. Configurable n-gram threshold (
ngramThreshold, default: 1). Safety: if both n-gram and LLM confirm are disabled, no oaths created.Configurable classifier model
Defaults to
letta/auto-fast(cheap, fast). Set per-conversation when creating throwaway conversations.TUI Dashboard
Standalone Rust binary (ratatui + crossterm):
Self-healing port discovery
The mod verifies the server port is alive every 60s. If dead (app restarted), falls back to ss discovery. Survives app restarts without manual /reload.
Aggressive pruning
prefilter_rejectedpruned after 10 minutesfalse_positivepruned after 30 minutesdelivered/failedpruned after 24 hoursTest plan
Generated with Letta Code