Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,8 @@ Master-skill 的核心不是"角色扮演提示词集合",而是一个可验
| 已测通过 / 已测总数 | **59 / 84(70%)** |
| 全量夹具覆盖率 | 84 / 211(40%)—— 运行途中 API 账户余额耗尽(HTTP 400),非限流也非代码缺陷,剩余 127 条**未测**,不计入失败 |
| 真实失败聚集 | 关键词未覆盖 14/25;禁用词命中 12/25;引用缺失 5/25;**零虚构引用** |
| ⚠️ 量具告警 | 12 条禁用词失败里有 **10 条,禁用词本就出现在提问中**(陷阱题),而检查是对回答做纯子串匹配——正确的驳斥与真正的越界会被同等判失败。**故 70% 是下限而非估计值**,详见基线报告 |
| 分测试类型 | 常规教理 43/48 = **89.6%**;守边界 12/26 = **46.2%**;抗压守引用 4/10 = **40.0%** —— 弱项是护栏,不是教理内容 |
| ⚠️ 量具告警 | 12 条禁用词失败里有 **10 条,禁用词本就出现在提问中**(陷阱题),而检查是对回答做纯子串匹配——正确的驳斥与真正的越界会被同等判失败。**故真实值落在 70.2%–75.0%**(10 条中仅 4 条剔除该项后会翻成 PASS,其余 6 条另有检查未过),详见基线报告 |

这是**关键词/引用字符串覆盖率检查,不是教义正确性或 LLM 判分的答案质量**。完整表格、失败案例与方法论说明见 **[eval/reports/BASELINE.md](eval/reports/BASELINE.md)**。

Expand Down
3 changes: 2 additions & 1 deletion README_EN.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,8 @@ The 211 fixtures under `tests/fidelity.jsonl` used to be just fixtures — `scri
| Passed / measured | **59 / 84 (70%)** |
| Coverage of the full suite | 84 / 211 (40%) — the run stopped partway when the API account's credit balance ran out (HTTP 400), not from rate limiting or a code bug; the remaining 127 cases were never evaluated and are **not** counted as failures |
| Where real failures cluster | Missing expected keyword 14/25; forbidden phrase present 12/25; missing citation 5/25; **zero fabricated citations** |
| ⚠️ Instrument warning | 10 of the 12 forbidden-phrase failures name a term **that already appears in the question** (these are trap questions), and the check is a plain substring match on the response — a correct refusal fails it exactly as hard as a real violation. **So 70% is a floor, not an estimate.** See the baseline report |
| By test type | ordinary doctrine 43/48 = **89.6%**; boundary 12/26 = **46.2%**; citation-under-pressure 4/10 = **40.0%** — the weakness is in the guardrails, not the doctrinal content |
| ⚠️ Instrument warning | 10 of the 12 forbidden-phrase failures name a term **that already appears in the question** (these are trap questions), and the check is a plain substring match on the response — a correct refusal fails it exactly as hard as a real violation. **So the true rate is in [70.2%, 75.0%]** — only 4 of the 10 would flip; the other 6 fail on independent checks. See the baseline report |

This measures **keyword/citation-string coverage, not doctrinal correctness or LLM-judged answer quality**. Full table, failing cases, and methodology notes: **[eval/reports/BASELINE.md](eval/reports/BASELINE.md)**.

Expand Down
30 changes: 28 additions & 2 deletions eval/reports/BASELINE.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,8 +105,12 @@ A correct refusal — "你问『禅宗最究竟』,这个提法本身就把宗
exactly as hard as an actual ranking does. So does merely restating what was asked. Only
**2 of the 12** are unambiguous violations; the other 10 are undecidable from this run.

**That makes 70.2% a floor, not an estimate.** Up to 10 of the 25 failures (40%) may be
measurement artifacts rather than persona defects.
**That makes 70.2% a floor — but a tight one.** Of the 10 undecidable cases, only **4**
would flip to PASS if the forbidden-phrase check excluded question terms; the other 6 fail
independently on `must_mention` or `must_cite` and would stay failures either way. So the
true rate sits in **[70.2%, 75.0%]**, a 4.8pp band. The instrument defect makes individual
verdicts unauditable — it does not make the headline number soft. Most of these failures
are real.

### Measurement limitation: responses are not persisted

Expand Down Expand Up @@ -154,6 +158,28 @@ resolved to a real declared source. Source-grounding against hallucinated citati
up in this sample; it's the "don't rank traditions" boundary and "keep citing under
pressure" behaviors that show real gaps.

## The cut that matters: by test type

Aggregate pass rate hides the actual signal. Split the 84 measured cases by fixture
`test_type`:

| test_type | Passed | Measured | Pass rate | What it tests |
|---|---:|---:|---:|---|
| `fidelity` | 43 | 48 | **89.6%** | ordinary doctrinal Q&A — citations and keyword coverage |
| `boundary` | 12 | 26 | **46.2%** | refuse to rank traditions, stay inside the school, no attainment prediction |
| `pressure` | 4 | 10 | **40.0%** | keep citing when the user explicitly asks you to stop |

Even taking the instrument caveat at its most generous (all 4 flippable cases are boundary
ones, giving 16/26 = 61.5%), boundary remains far below fidelity. The `pressure` cluster
carries no instrument caveat at all — those failures are clean `must_cite` / `must_mention`
misses.

**This inverts the project's own self-description.** The README leads with four pillars:
source-grounded, boundary-aware, fidelity-tested, runtime-ready. The measurement says the
persona *content* works (89.6%, zero fabricated citations) and the *guardrails* do not
(46.2% / 40.0%) — and the guardrails are what `ETHICS.md` exists to guarantee. Boundary
behaviour, not doctrinal accuracy, is where this project's measured risk lives.

## What we did not change

Per this task's scope: no `fidelity.jsonl` fixture was edited, and no pass/fail threshold
Expand Down
Loading