docs(eval): commit first real fidelity baseline (59/84 measured, 40% coverage) - #130
Merged
Conversation
Previously scripts/test-fidelity.py only printed to stdout; the 211 fidelity.jsonl fixtures had never actually been scored and recorded, so the "Fidelity-tested" pillar was unmeasured. Ran the full graded suite (--all --json, claude-sonnet-4-6) against commit c697d5d. The run stopped partway when the API account's credit balance ran out (HTTP 400), not from rate limiting or a code defect: 84/211 fixtures got a real graded response (59 passed, 70%), the remaining 127 were never evaluated and are recorded as "not measured" rather than failures. Adds: - eval/reports/0.10.1-c697d5d.json — raw run output (verbatim suites array from the script) wrapped in a meta block with the measured commit, model, timestamps, and coverage/pass totals. - eval/reports/BASELINE.md — human-readable table plus an honest breakdown of what the real failures cluster around (boundary tests leaking comparative language, citations dropped under "don't cite" pressure, missing keyword coverage; zero fabricated citations). - eval/reports/README.md — what this directory is, how to regenerate, and what the numbers do/do not measure. Updates README.md / README_EN.md with a compact, truthful summary of the real numbers near the existing pillar claims, linking to eval/reports/BASELINE.md for detail. No fidelity.jsonl fixture or scoring logic was changed.
The first baseline attributed its largest failure cluster to the personas:
14 boundary cases "leaking comparative language under pressure." Re-reading
the raw results against the fixtures does not support that.
`must_not_contain` is a plain substring match on the response, and these
boundary fixtures are baited questions that carry the loaded term themselves.
Of the 12 failures that tripped it, 10 name a term already present in the
question — 更好 / 更接近 / 最高 / 神通 / 禅宗最究竟 / 往生净土 / 八识 …
A correct refusal ("你问『禅宗最究竟』，这个提法本身就把宗派当成了名次")
fails the check exactly as hard as a real ranking would. So does restating
the question. Only 2 of the 12 are unambiguous violations.
So 70.2% is a floor, not an estimate: up to 10 of the 25 failures may be
artifacts of the instrument. Worse, this cannot be settled from the run —
test-fidelity.py persists only response_length, never the response, so no
failure is adjudicable after the fact. Both are now written up as findings,
with two non-question-begging fixes proposed (persist responses; exclude
terms that occur in the fixture's own question). No fixture and no threshold
was changed.
Also corrected: the pressure cluster is 6 failures of which 4 dropped a
citation, not 6 dropping citations.
Unrelated doc debts fixed while in the file:
- desktop manager manages 19 skills, not 17
- the screenshot URL pointed at branch `master`, which does not exist; it
rendered only via GitHub's legacy default-branch redirect. Now `main`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The README advertises
Fidelity-testedas one of four pillars, but no fidelity score hadever actually been committed to the repo —
scripts/test-fidelity.pyonly printed tostdout, and CI only ran
--dry-run(structural check, no API calls). This PR runs thegraded suite for real and commits the first honest baseline.
c697d5d3be78ce6738cf1f969ca057c7e4c16bb5claude-sonnet-4-6The run stopped partway through when the Anthropic account's API credit ran out
(
HTTP 400 — Your credit balance is too low), not from rate limiting or a bug — theharness itself worked correctly (confirmed with a smoke test first, and every rejected
call was caught and recorded as
api_errorinstead of crashing the run). The remaining127 fixtures (
master-huinengthroughmaster-zhiyi, plus 8/10 ofmaster-help) werenever evaluated and are recorded as not measured, not as failures.
leaks comparative language under direct "which is better" pressure (14/25 failures),
citations dropped when the user explicitly asks not to cite scripture (6/25), and missing
expected keywords (5/25). Zero fabricated citations in the measured sample.
New files:
eval/reports/0.10.1-c697d5d.json— raw run output (verbatimsuitesarray from thescript), wrapped in a
metablock with commit, model, timestamps, coverage/pass totals.eval/reports/BASELINE.md— full table + honest failure-cluster analysis.eval/reports/README.md— what the directory is, how to regenerate, what the numbersdo/do not mean, and cost.
README.md/README_EN.mdget a compact, truthful summary near the existing pillarclaims, linking to
eval/reports/BASELINE.mdfor detail.Not changed: no
fidelity.jsonlfixture, no pass/fail threshold or checking logic inscripts/test-fidelity.py, andpackage.json'sfiles[](theeval/directory is notshipped in the npm tarball).
中文说明
README 首屏一直宣称四大支柱之一是「可评测」（Fidelity-tested），但仓库里从未提交过一次真
实评分——
scripts/test-fidelity.py只把结果打印到终端，CI 也只跑--dry-run（结构校验，不调用 API）。这个 PR 把 211 条
fidelity.jsonl夹具真正跑了一遍，并提交了第一份诚实基线。c697d5d3be78ce6738cf1f969ca057c7e4c16bb5claude-sonnet-4-6跑到一半 Anthropic 账户 API 余额耗尽（HTTP 400），不是限流也不是代码缺陷——harness 本身工作
正常（先用冒烟测试确认过），每个被拒绝的请求都被正确捕获记为
api_error而不是让整个跑批崩溃。剩余 127 条夹具（
master-huineng到master-zhiyi，以及master-help的 8/10）从未被实际评测，记为未测，不计入失败。
（14/25）；用户明确要求"别引经据典"时引用被一并省略（6/25）；预期关键词缺失（5/25）。已测
样本中零虚构引用。
新增文件：
eval/reports/0.10.1-c697d5d.json—— 脚本输出的原始suites数组（逐字保留），外层包一个
meta块（commit、模型、时间戳、覆盖率/通过率汇总）。eval/reports/BASELINE.md—— 完整表格 + 诚实的失败聚类分析。eval/reports/README.md—— 目录说明、如何重跑、这些数字代表什么/不代表什么、成本。README.md/README_EN.md在现有支柱声明附近加了一段简短、真实的数字摘要，并链接到eval/reports/BASELINE.md。未改动： 未修改任何
fidelity.jsonl夹具，未改动scripts/test-fidelity.py的通过/失败阈值或判定逻辑，
package.json的files[]未变更（eval/目录不会进 npm 发布包）。Test plan
python3 scripts/test-fidelity.py --master master-yinguang --max-tests 1— smoke test, 1/1 PASS, confirms the harness works end to endpython3 scripts/test-fidelity.py --all --json --model claude-sonnet-4-6— full graded run, ran ~44 minutes before hitting API credit exhaustion; 84/211 fixtures measured (59 passed)npm test— 60/60 passingpython3 -m pytest -q— 373 passedpackage.json'sfiles[]unchanged (eval/not shipped in the npm tarball)Follow-up
Re-run
python3 scripts/test-fidelity.py --all --json --model claude-sonnet-4-6once theAPI key has credit again to cover the 127 fixtures this run never reached, and update
eval/reports/with a full 211/211 baseline.