Skip to content

ConTrust: credibility-weighted context priors (streamlined) - #34

Open
muhaiminsk wants to merge 5 commits into
IBM:mainfrom
muhaiminsk:contrust-minimal
Open

ConTrust: credibility-weighted context priors (streamlined)#34
muhaiminsk wants to merge 5 commits into
IBM:mainfrom
muhaiminsk:contrust-minimal

Conversation

@muhaiminsk

Copy link
Copy Markdown

Streamlined branch as discussed — the prior classification and consensus
scoring only, so this can be reviewed and merged independently of the
research branch (#33).

FactReasoner enters every retrieved context into the Markov network at a
fixed PRIOR_PROB_CONTEXT = 0.9. This adds an optional per-source weight:

w = (1 − β)·prior + β·r

prior  published media-credibility rating (MBFC), exact-TLD rule for
       .gov/.edu/.int/.mil → 0.90, unrated → 0.50 neutral
r      (1 + agreed)/(2 + seen) — Beta posterior mean over agreement with
       the credibility-weighted consensus of the other evidence
β      min(a/2, 0.7) — the prior dominates until a source has a record

No existing file is modified. The scorer sets context.probability
between pipeline.build() and pipeline.score(), which the assessor
already reads. It does not touch the NLI path — it only reads
relation.probability after the fact.

Five files, all new:

  • src/fact_reasoner/core/contrust.py (234 lines)
  • data/priors/mbfc_idiap.csv — MBFC ratings via idiap (Apache-2.0),
    Sánchez-Cortés et al., CLEF 2024
  • tests/test_contrust.py — 5 tests
  • docs/examples/assessors/ex_factreasoner_contrust.py — the existing
    example with a 3-line insertion
  • examples/contrust_example.py — standalone demo, no API required

Evaluations, ablations, baselines and frozen reproduction artifacts are on
feature/consensus-trust (#33).

muhaiminsk and others added 5 commits July 31, 2026 06:33
Replaces FactReasoner's fixed PRIOR_PROB_CONTEXT with a per-source weight
w = (1-beta)*prior + beta*r, where prior is a published media-credibility
rating and r is a Beta posterior mean over agreement with the
credibility-weighted consensus of other evidence. No gold labels are used
and no changes to the inference code are required.

Prior data: idiap/Factual-Reporting-and-Political-Bias-Web-Interactions
(Apache-2.0), Sanchez-Cortes et al., CLEF 2024.

Signed-off-by: Samit Muhaimin <muhaiminsamit@gmail.com>
Signed-off-by: Samit Muhaimin <Samit.Muhaimin@partner.ibm.com>
Shows credibility weighting applied to IBM's own ex_factreasoner.py: the
only difference is a block between pipeline.build() and pipeline.score()
that sets each context's probability from its source's credibility, plus
a call to update the source records afterwards. No FactReasoner code is
modified. Also drops an unused argument from update_from_results().

Signed-off-by: Samit Muhaimin <muhaiminsamit@gmail.com>
Signed-off-by: Samit Muhaimin <Samit.Muhaimin@partner.ibm.com>
FactReasoner.build() is a coroutine; the upstream example calls it
synchronously, which silently builds nothing and makes score() assert.
Wrapped in asyncio.run(). Verified end-to-end on the Lanny Flaherty
example: 15 atoms, 40 contexts, 151 relations, ~18 min via RITS.

Signed-off-by: Samit Muhaimin <Samit.Muhaimin@partner.ibm.com>
Signed-off-by: Samit Muhaimin <Samit.Muhaimin@partner.ibm.com>
Clarified limitations of the scoring system and improved explanation of prior score generation.

Signed-off-by: Samit Muhaimin <Samit.Muhaimin@partner.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant