Skip to content

RLCR: BitLesson knowledge base rots silently — add a staleness/reference check + a deprecate action #186

@LukeLIN-web

Description

@LukeLIN-web

Summary

RLCR's BitLesson knowledge base (.humanize/bitlesson.md) is effectively append-only.
The round stop-gate validates the format of the ## BitLesson Delta (per docs/bitlesson.md:
Action ∈ none|add|update, IDs must exist), but nothing validates that existing lessons
are still accurate against the current repo
. As the codebase refactors, lessons rot —
and a rotted lesson handed to an implementer by bitlesson-select.sh is worse than no
lesson, because it actively misleads.

Where it breaks (observed in a real knowledge base after one dir reorg)

A subsystem's code moved eval_audio_cot/scripts/scripts/audio_cot/ (output
namespace stayed outputs/eval_audio_cot/). All of the following passed the format-only gate:

  • 3 lessons' Scope: lines still point at the old eval_audio_cot/scripts path (dir gone).
  • A Solution: snippet says derive root via Path(__file__).resolve().parent.parent,
    but the file is now nested one level deeper, so current code uses parent.parent.parent
    — copying the lesson literally is off-by-one.
  • A canonical path outputs/analysis/{score,go_nogo_decision}.json moved to
    outputs/eval_audio_cot/analysis/....
  • Update (round 9) notes superseded by Update (round 22) linger with no way to retire them.

Root cause

  • The Delta contract has no deprecate/delete action (only none|add|update), so
    superseded lessons and stale Update (round N) notes can only be piled on, never retired.
  • No step ever re-checks that a lesson's cited paths / file:line refs / code snippets
    still resolve. (Cf. the agent-memory rule: "a memory naming a file is a claim it existed
    when written — verify before acting." BitLessons make the same claims but are never re-verified.)

Proposed improvements

  1. Add deprecate (and/or delete) to the BitLesson Delta (Action: none|add|update|deprecate),
    with provenance ("superseded by BL-… / dir reorg"). Lets the loop retire dead lessons
    and consolidate old Updates.
  2. A reference-resolution / staleness check — companion to bitlesson-select.sh, run at
    loop start or as a periodic gate — that parses each lesson's Scope paths + file:line
    refs and reports entries whose references no longer exist. Advisory by default; optionally
    blocking via a flag (like the existing --require-bitlesson-entry-for-none).
  3. Selector surfaces staleness at point of use: when a lesson is selected, re-validate its
    referenced paths; if any don't resolve, tag it "possibly stale — verify before applying".
  4. Update-note curation: when an entry accrues many Update (round N) lines, prompt to fold
    them into the Solution and drop superseded ones.

Scope / non-goals

Lesson content (the bug→fix knowledge) usually stays valid across refactors; only its
references drift. Default should be fix/flag references, not delete whole lessons —
deletion reserved for genuinely obsolete subsystems via the explicit deprecate action.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions