docs(eval): split the baseline by test type, tighten an overstated caveat - #131
Merged
Conversation
… overstated
Two corrections to the baseline write-up.
1. The caveat was too generous to itself. It said "up to 10 of the 25
failures (40%) may be measurement artifacts." True but misleading: of
those 10 undecidable cases only 4 would flip to PASS if the
forbidden-phrase check excluded question terms — the other 6 also miss a
must_mention or must_cite and fail either way. The band is
[70.2%, 75.0%], 4.8pp. The instrument defect makes individual verdicts
unauditable; it does not make the headline soft. Most failures are real.
2. The aggregate hid the actual signal. By fixture test_type:
fidelity 43/48 = 89.6% ordinary doctrinal Q&A
boundary 12/26 = 46.2% don't rank traditions / stay in school
pressure 4/10 = 40.0% keep citing when told to stop
This inverts the project's own framing. The persona content works — 89.6%
plus zero fabricated citations. The guardrails do not. And the guardrails
are what ETHICS.md exists to guarantee: no sectarian ranking, no crossing
into another school's doctrine, no attainment prediction.
Even granting every flippable case to boundary (16/26 = 61.5%), it stays
far below fidelity. The pressure cluster has no instrument caveat at all.
Measured risk in this project is in boundary behaviour, not doctrinal
accuracy. No fixture and no threshold was changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
中文摘要
对上一份基线（#130）的两处修正，都是把话说准，没有改任何 fixture 或阈值。
一、我上一版的免责说法太宽松。 原文写「25 条失败里最多 10 条可能是量具问题（40%）」——技术上没错但会误导。那 10 条不可裁定的样本里，只有 4 条在禁用词检查排除题干词之后会翻成 PASS；其余 6 条另有
must_mention/must_cite未过，怎么算都是失败。所以真实区间是 [70.2%, 75.0%]，宽度 4.8pp。量具缺陷让单条判定不可审计，但没有让总分变软——这些失败大部分是真的。二、总分掩盖了真正的信号。 按 fixture
test_type拆开：fidelityboundarypressure这与项目的自我描述正好相反。 README 的四根柱子是「有来源 · 守边界 · 可评测 · 可运行」。实测说：人格的内容是立得住的（89.6%，零虚构引用）；护栏不行（46.2% / 40.0%）。而护栏正是
ETHICS.md存在的理由——不判宗派高下、不越宗说法、马哈希那条不得对个体作证果判定。即便把 4 条可翻转样本全算给 boundary（16/26 = 61.5%），它仍远低于 fidelity。
pressure那一簇则完全没有量具争议，失败干净利落。结论：这个项目已测出的风险在守边界，不在教理准确性。