Summarize CBDR/LCR papers and draft confidence-aware reranking spec#7
Open
pko89403 wants to merge 15 commits into
Open
Summarize CBDR/LCR papers and draft confidence-aware reranking spec#7pko89403 wants to merge 15 commits into
pko89403 wants to merge 15 commits into
Conversation
Reference digests written strictly from the committed PDFs, with page- cited numbers and explicit '논문에 명시되지 않음' markers for gaps (including the CBDR paper's internal inconsistencies: Mid_Layer vs hidden_states[-1], dataset-size and epoch discrepancies). - docs/wiki/references/cbdr_parametric_confidence_rag.md: CBDR requires white-box hidden-state access at train and inference, so only its design pattern transfers to ranksmith; retrieval triggering is out of library scope - docs/wiki/references/llm_confidence_reranker.md: training-free MSCP reranking; conflicts with the temperature-0 JSON-only ModelClient contract are recorded - docs/specs/spec_confidence_aware_reranking.md: Draft spec consuming the runtime-readiness signal contract; five open decisions filed as Q004 in 05_open_questions.md, no implementation before resolution - References index updated; stale metrics path fixed in spec TEMPLATE Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Q004-1 resolved: confidence source = structural + LightGBM scorer (path A), chosen over LCR/MSCP because MSCP needs K+ LLM calls per document. Smoke run (LM Studio qwen3.5-9b, libomp installed) confirmed: - generation pipeline runs end-to-end (15 fixture rows generated) - FrozenAutoEncoder loads and extract_structural_features produces the documented 70-dim structural-v1 vector on real data - training is blocked only by the MIN_TOTAL_SAMPLES=30 guard; the repo fixture yields 15, so producing an artifact needs >=30 real labels Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fetched real BEIR/SciFact data (56 samples: 14 qrels positives + 42 lexical hard negatives), generated judgments via LM Studio, extracted bert features, trained LightGBM + sigmoid calibration, and produced a real scorer.joblib that scores held-out test features. Two findings recorded in Q004: 1. load_lightgbm_scorer misroutes: when metadata_path is passed it assumes a raw Booster file, but the training pipeline exports a joblib dict. The spec's documented from_artifact(path, metadata_path=) call fails on the pipeline's own artifact; from_artifact(path) works. Same root cause as YAGNI finding #1 (4 loader formats, 1 produced). 2. macOS ARM: torch + lightgbm in one process deadlock/segfault via dual OpenMP; artifact was produced by splitting extraction and training into separate processes. Not a ranksmith code bug. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Confidence source = path A (structural confidence + LightGBM); form = new Strategy; ranking = signed confidence. Confidence is computed locally (frozen encoder + LightGBM scorer) with zero LLM calls — only one relevance judge() call per document, never LCR-style repeated sampling. - model.py: add ModelClient.judge / AsyncModelClient.judge (relevance verdict), mirroring rank/compare/select - parsing.py: add parse_judgment_response - strategies/confidence.py: ConfidenceRerankStrategy / AsyncConfidenceRerankStrategy. Per doc: judge -> structural confidence -> signed score (+conf if relevant, -conf if not); sort desc, stable on ties. Estimator typed via Protocol so strategies never force a torch import. Async gathers judge calls concurrently, scores sequentially. - Exported from ranksmith and ranksmith.strategies. Verified: 8 unit tests (synthetic judge + fake estimator) + end-to-end against LM Studio judge with the smoke artifact (relevant docs sort to the top, 3/3). 440 tests pass, mypy + ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the earlier signed-judgment reranker with the CBDR-faithful confidence-change approach. CBDR ranks documents by how much each one raises the model's answer confidence: Inc = C(query+doc) - C(query). Within a single query the baseline C(query) is a constant (and answer_confidence cannot score an empty context anyway), so ordering by C(query+doc) is exactly the confidence-change ranking. - model.py: replace judge() with answer(query, context) on both clients, using the same answer prompt as the generation pipeline so training and inference stay consistent - parsing.py: replace parse_judgment_response with parse_answer_response - strategies/confidence.py: AnswerConfidenceRerankStrategy / AsyncAnswerConfidenceRerankStrategy. Per doc: answer -> local answer_confidence -> sort desc. Estimator typed via Protocol; async gathers answer calls, scores sequentially. Confidence uses zero LLM calls; one answer call per document, never LCR-style repeated sampling. Verified: 8 unit tests + end-to-end against LM Studio with an answer_confidence artifact trained on 60 SQuAD samples — the gold context sorts to rank 1 (magnitudes cluster; the smoke artifact discriminates weakly, as expected). 440 tests pass, mypy + ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Held-out gold-context recovery over 15 SQuAD cases (gold + 3 random distractors), answer_confidence trained on LM Studio judgments: samples test_auc rerank_acc@1 MRR 100 0.333 0.000 0.294 500 0.875 0.800 0.878 random 0.5 0.250 0.521 Data scale is the deciding factor: 100 samples overfit (worse than random, gold sinks to the bottom); 500 samples discriminate strongly (gold ranks #1 in 12/15). Honest caveats recorded: distractors are unrelated SQuAD contexts (easy signal via NO_ANSWER), eval set is small, and no README performance claim is made. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ponytail review found the answer prompt duplicated byte-for-byte in model._answer_messages (reranker inference) and confidence_generation.build_answer_prompt (training data generation). They must stay identical for train/inference consistency, but nothing enforced it, and generation's configurable no_answer_value could silently diverge from the reranker's. Make model._answer_messages the single builder (now takes no_answer_value, default '__NO_ANSWER__'); the generation pipeline calls it. Delete ANSWER_SYSTEM_PROMPT and build_answer_prompt from confidence_generation. Tests updated to the shared builder. 440 tests pass, mypy + ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the earlier prompt unification. The reranker hardcoded the NO_ANSWER sentinel while AnswerGenerationConfig still let generation set a different no_answer_value — so a custom-sentinel artifact would silently mismatch the reranker's inference prompt. My earlier 'consistency guaranteed by code' claim was therefore false. Fix the sentinel to the single NO_ANSWER_VALUE constant (model.py): - drop no_answer_value from _answer_messages, AnswerGenerationConfig, and normalized_exact_match; all use the constant - generation labeling/metadata source it from the same constant Verified by reflection that no divergence path remains (no no_answer parameter on any of the three). 439 tests pass, mypy + ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same 15 held-out cases, ConfidenceRerank (500-sample artifact) vs ListwiseStrategy: strategy acc@1 MRR calls/query ConfidenceRerank (500) 0.800 0.878 4.0 ListwiseStrategy 1.000 1.000 1.0 Listwise is perfect and 4x cheaper on this easy setting. No setting has yet shown the confidence reranker beating an existing strategy. Recorded honestly; usefulness is not oversold. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
examples/answer_confidence.py mirrors the other fake-provider examples (deterministic KeywordAnswerClient + a keyword estimator standing in for StructuralConfidenceEstimator.from_artifact). test_examples runs it without a live provider and pins the output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Audit found the new strategy was missing from the conventional surfaces that every other strategy covers, and that several docs still said 'confidence is not a reranker' (now false). - Fix the now-false statements: the confidence estimator is a scoring utility; AnswerConfidenceRerankStrategy consumes it and IS a reranking Strategy (guardrails.md, SKILL.md, README x2, 02_architecture.md) - README.md/README.ko.md: add an 'Answer Confidence Reranking (experimental)' section — honestly notes it loses to Listwise (acc@1 0.80 vs 1.00) at 4x cost and needs a trained artifact - advisor method-guide.md: quick-decision row + parameters entry, marked experimental / not-a-default with the Listwise baseline named - snippets.md: reference examples/answer_confidence.py - test_advisor_references: guard AnswerConfidenceRerankStrategy defaults - docs/wiki/00_context.md: list the new strategies 440 tests pass, mypy + ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds 'answer_confidence' as a selectable algorithm in scripts/compare_reranking.py so it compares against the other strategies on the standard qrels benchmark. Opt-in only (excluded from --algorithm all): needs --answer-confidence-artifact pointing at a trained scorer. Estimator is loaded once (lru_cache); nominal call estimate = one answer call per document. Records the special-case caveat (in CLI help and the spec): this IR benchmark has qrels but no gold answers, so it cannot train an answer_confidence scorer — you must supply one trained separately on QA data. Passing an artifact trained on another domain (e.g. SQuAD) measures domain shift, not a fair comparison, and must be labeled as such. The script targets the Azure SDK, so it does not run against LM Studio here; scripts/ is outside the mypy CI gate. 440 tests pass (30 compare_reranking), ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ere reality - compare_reranking.py: RANKSMITH_OPENAI_BASE_URL (+_MODEL/_API_KEY) points the run at any OpenAI-compatible endpoint (LM Studio, vLLM, ...) instead of Azure, for local/test runs. Azure stays the default when unset. - Spec: record that the README AskUbuntu benchmark cannot be reproduced in this dev env (gitignored corpus cache, Azure VNet-blocked, and AskUbuntu has no gold answers to train an answer_confidence artifact). It must be run on an environment that has the AskUbuntu corpus + Azure access. - Spec: record the preliminary LM-Studio/SciFact diagnostic (answer_confidence 0.311 NDCG@5 vs listwise 1.000 at 10x cost) — explicitly NOT for the README table (domain-shifted artifact, easy distractors, small local model). 440 tests pass, ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prepares the ground for the confidence-aware reranking (CBDR-direction) work by converting the two pending confidence papers into repo reference knowledge, per
docs/wiki/03_reference_processing.md.Fact discipline: both digests were written strictly from the committed PDF texts. Every number is page-cited in the extraction notes; anything the papers do not state is marked "논문에 명시되지 않음" instead of being filled in. The CBDR paper's own internal inconsistencies (Mid_Layer vs
hidden_states[-1], 1k/300/500 vs 2,000/1,000/500 dataset sizes, 30 vs 100 epochs) are recorded verbatim rather than reconciled.cbdr_parametric_confidence_rag.md— CBDR (anonymous ACL submission). Key finding: the method needs white-box hidden-state access at both training (74,204 target-LLM forward passes) and inference, so it cannot be ported to ranksmith's closed-model premise; only the "confidence signal drives document preference" pattern transfers. CBDR itself is a retrieval trigger, not a reranker — out of library scope.llm_confidence_reranker.md— LCR (arXiv:2602.13571). Training-free, genuinely black-box (sampled text only), so it fits ranksmith better; recorded conflicts: needs temperature-1 sampling K=10 vs our temperature-0 JSON-only ModelClient contract, and the paper never discloses its tuned thresholds or call-cost totals.spec_confidence_aware_reranking.md— Draft spec that consumes the existing runtime-readiness signal contract (order-preservingscore_batch+ caller-side candidate zip). Five design decisions are deliberately left open and filed as Q004 in05_open_questions.md; the spec states implementation must not start before they are resolved.src/ranksmith/_metrics.pypath in spec TEMPLATE fixed tobenchmarks/metrics.py.Docs-only change — no runtime code touched.
test_advisor_references.pydoc guards pass.🤖 Generated with Claude Code