ConTrust: credibility-weighted context priors (streamlined) - #34
Open
muhaiminsk wants to merge 5 commits into
Open
ConTrust: credibility-weighted context priors (streamlined)#34muhaiminsk wants to merge 5 commits into
muhaiminsk wants to merge 5 commits into
Conversation
Replaces FactReasoner's fixed PRIOR_PROB_CONTEXT with a per-source weight w = (1-beta)*prior + beta*r, where prior is a published media-credibility rating and r is a Beta posterior mean over agreement with the credibility-weighted consensus of other evidence. No gold labels are used and no changes to the inference code are required. Prior data: idiap/Factual-Reporting-and-Political-Bias-Web-Interactions (Apache-2.0), Sanchez-Cortes et al., CLEF 2024. Signed-off-by: Samit Muhaimin <muhaiminsamit@gmail.com> Signed-off-by: Samit Muhaimin <Samit.Muhaimin@partner.ibm.com>
Shows credibility weighting applied to IBM's own ex_factreasoner.py: the only difference is a block between pipeline.build() and pipeline.score() that sets each context's probability from its source's credibility, plus a call to update the source records afterwards. No FactReasoner code is modified. Also drops an unused argument from update_from_results(). Signed-off-by: Samit Muhaimin <muhaiminsamit@gmail.com> Signed-off-by: Samit Muhaimin <Samit.Muhaimin@partner.ibm.com>
FactReasoner.build() is a coroutine; the upstream example calls it synchronously, which silently builds nothing and makes score() assert. Wrapped in asyncio.run(). Verified end-to-end on the Lanny Flaherty example: 15 atoms, 40 contexts, 151 relations, ~18 min via RITS. Signed-off-by: Samit Muhaimin <Samit.Muhaimin@partner.ibm.com>
Signed-off-by: Samit Muhaimin <Samit.Muhaimin@partner.ibm.com>
Clarified limitations of the scoring system and improved explanation of prior score generation. Signed-off-by: Samit Muhaimin <Samit.Muhaimin@partner.ibm.com>
muhaiminsk
force-pushed
the
contrust-minimal
branch
from
July 31, 2026 10:34
5557ee9 to
1145361
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Streamlined branch as discussed — the prior classification and consensus
scoring only, so this can be reviewed and merged independently of the
research branch (#33).
FactReasoner enters every retrieved context into the Markov network at a
fixed
PRIOR_PROB_CONTEXT = 0.9. This adds an optional per-source weight:No existing file is modified. The scorer sets
context.probabilitybetween
pipeline.build()andpipeline.score(), which the assessoralready reads. It does not touch the NLI path — it only reads
relation.probabilityafter the fact.Five files, all new:
src/fact_reasoner/core/contrust.py(234 lines)data/priors/mbfc_idiap.csv— MBFC ratings via idiap (Apache-2.0),Sánchez-Cortés et al., CLEF 2024
tests/test_contrust.py— 5 testsdocs/examples/assessors/ex_factreasoner_contrust.py— the existingexample with a 3-line insertion
examples/contrust_example.py— standalone demo, no API requiredEvaluations, ablations, baselines and frozen reproduction artifacts are on
feature/consensus-trust(#33).