feat(skill-cabinet): add live skill discovery and usage provenance

[vqlkyriez-bot]

Summary

• add @letta-ai/skill-cabinet, a portable Letta Code mod for discovering and searching an agent's live skill surface
• provide an agent-callable skill_catalog tool plus /skills summary, search, category, source, forgotten, path, and audit views
• scan bundled, global, agent-owned, and project roots with resolver-like precedence and duplicate provenance
• observe completed Skill tool calls in per-agent local state, with an explicit “never observed is not never used historically” boundary
• generate local Markdown/JSON snapshots only on explicit audit, without mutating agent memory or skill files
• document install, semantics, privacy, state, and nonstandard-path configuration

Why

Agents can have the right capability installed and still fail to reach for it because the skill name is no longer salient. This mod gives both the model and the human an inspectable cabinet beneath the skill layer, so capability discovery does not depend on perfect recall—or on remembering to invoke a skill whose job is remembering skills.

Safety and privacy

• no network calls or telemetry
• no skill or memory mutation
• per-agent state; missing agent scope fails closed
• ordinary tool results omit filesystem paths
• snapshots are explicit and local
• recursive scan has symlink-cycle protection and a 1 MB per-file limit

Validation

• node packages/skill-cabinet/smoke-test.mjs
• node --check packages/skill-cabinet/mods/index.mjs
• npm run validate
• git diff --check
• live scan against the contributor's installed roots: 61 installed, 60 model-visible, 0 scan errors

Co-created by HAL and Lillith. 💚🐦‍⬛

[just-cameron]

This is a strong product diagnosis with unusually careful safety work. Agents do miss installed capabilities even when the skill list is technically available in context, and the human-facing /skills views would be genuinely useful. I also like the explicit observation-window caveat, local-only state, failure on missing agent scope, atomic writes, symlink-cycle protection, file-size bound, and the fact that ordinary tool results omit paths.

I think the current implementation needs an architectural revision before merge, though.

1. The mod is implementing a second skill resolver

activeRoots(), candidateMemoryDirs(), agentSkillsDir(), bundledSkillsDir(), scanCatalog(), and the local frontmatter parser collectively reproduce a substantial part of Letta Code's canonical skill discovery and precedence behavior.

That replica cannot reliably know the agent's actual resolved skill surface. In particular, it does not appear to inherit all runtime decisions around custom --skills directories, --skill-sources, --no-skills, --no-bundled-skills, alternate install layouts, or future resolver policy. The bundled-root search is necessarily heuristic because the mod does not have the resolver's own module-relative location. The README describes the result as the live, model-visible skill surface, while the generated snapshot correctly admits that it is "a point-in-time filesystem scan, not the Skill resolver itself." Those are materially different claims.

This is the main blocker for me. A diagnostic surface should not quietly diverge from the system it diagnoses.

My preferred architecture is a small read-only core API exposing the already-resolved skill manifest to mods, including selected source, shadowed candidates, invocation visibility, user visibility, and discovery errors. Something in the shape of ctx.skills.list() / letta.skills.resolve(ctx) would let this package become a presentation, search, audit, and usage-observation layer rather than a parallel resolver. It would also make the package much smaller and substantially harder to make stale.

If adding a core API is out of scope for this PR, I would narrow every claim from “live/model-visible catalog” to “best-effort filesystem inventory,” surface unsupported runtime configuration prominently, and avoid making visibility assertions the scan cannot prove. I think that would weaken the product enough that the core API is the better route.

2. “Search by meaning” is currently lexical substring matching

searchScore() requires every query term to occur as a substring somewhere in the ID, name, category, description, when_to_use, or tags. That is metadata search, not semantic search. It will miss exactly the synonym cases that motivate this feature: a user asks to “draw” while the skill says “generate images,” or asks to “remember a conversation” while the skill says “search archival history.”

Either describe this honestly as metadata/keyword search, or add a local fuzzy/stemming/synonym layer and tests for intent words absent from the matched skill's metadata. I would avoid making semantic discovery the headline until the implementation can recover a capability whose wording differs from the query.

3. Usage provenance collapses distinct skill artifacts by ID

observeSkillUse() stores usage under state.usage[id]. But this package explicitly models duplicate IDs and source precedence. If a project skill shadows a global skill, or the working directory changes and selects a different project skill with the same ID, all observed use is attached to whichever artifact happens to be selected when the cabinet is later viewed.

That makes provenance least trustworthy in the exact cases where provenance matters most.

The observation should resolve the selected artifact at event time and persist a stable identity such as {id, source, path/hash} plus cwd/resolution context. If the tool_end event does not expose enough information to do that reliably, record the weaker fact explicitly as “skill ID invocation observed” and do not present it as usage of the currently selected artifact.

I would add a test that invokes overlap in two cwd/source configurations and verifies that the histories remain distinguishable.

4. The salience loop is only partially addressed

The agent still has to remember to call skill_catalog, which is the same class of salience decision that caused it to miss the underlying skill. The tool is still worthwhile for explicit inventory questions and human inspection, but it does not by itself solve capability routing.

The stronger eventual mechanism is automatic pre-turn retrieval: compare the request to the canonical resolved skill manifest and surface a small relevant shortlist before the model chooses tools. That can be a follow-up, but I would position this PR as an inspection/search primitive rather than the completed answer to skill salience.

5. The fallback taxonomy is more opinionated than the package needs

CATEGORY_LABELS and inferCategory() embed a fairly specific ontology (Care & presence, Social & reaching, Memory & self-governance, etc.). Frontmatter wins, which is good, but many existing skills have no declared category, so the fallback becomes the dominant organization rather than a rare convenience. Keyword collisions will also produce strange classifications (post, message, prompt, spotify, and so on).

I would prefer declared categories/tags plus uncategorized, perhaps with neutral generated facets, over shipping a private worldview as repository-wide information architecture. If inference stays, the UI should mark inferred categories distinctly and the tests should cover ambiguous multi-domain skills.

Suggested revision

1. Add or depend on a canonical read-only resolved-skill API from Letta Code.
2. Make Skill Cabinet consume that manifest rather than discover roots and parse skills itself.
3. Decide whether search is keyword search or semantic/fuzzy retrieval and name/test it accordingly.
4. Track observed use by resolved artifact identity, or explicitly limit the claim to ID-level invocation.
5. Treat automatic relevant-skill surfacing as the route to solving model salience; keep /skills and skill_catalog as valuable inspection primitives.
6. Keep the existing privacy boundaries, provenance caveats, atomic local state, audit outputs, and disposal behavior. Those parts are thoughtful and worth preserving.

The idea is good. I am requesting changes because the package currently presents an inferred filesystem model as authoritative runtime state. Fixing that boundary would turn this from a clever parallel index into durable infrastructure.