On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models

This is the official implementation of CalibSFT, a plug-and-play supervised fine-tuning stage that shapes a calibrated, broadly supported verbalized-confidence prior before confidence-aware reinforcement learning. For each training question, CalibSFT samples $n$ responses from the base model and labels each response with the confidence $c^* = \lambda q + (1-\lambda) z$, where $q$ is the question's success rate and $z$ is the response's correctness. It then balances the data across confidence levels from 0 to 1 and supervises confidence on all responses, but reasoning and answers only on correct ones. The resulting checkpoint initializes confidence-aware RL methods such as RLCR, CoCA, DCPO, and ReDoubt.

Installation

Please refer to docs/INSTALL.md to set up the environment.

Quick Start

We release the CalibSFT and CalibSFT → RLCR checkpoints for quick reproduction. Evaluate them on all 16 benchmarks (temperature 0.6; 4 responses per question, 32 on AIME):

Qwen3-8B

CKPT=Qwen/Qwen3-8B \
bash scripts/examples/eval/base.sh

CalibSFT

CKPT=SUSTech/Qwen3-8B-CalibSFT \
bash scripts/examples/eval/calib_sft.sh

CalibSFT → RLCR

CKPT=SUSTech/Qwen3-8B-CalibSFT-RLCR \
bash scripts/examples/eval/calib_sft_rlcr.sh

Training

Please refer to docs/RUN.md for CalibSFT data generation, CalibSFT training, confidence-aware RL, and evaluation of trained checkpoints.

Citation

@article{wang2026pitfalls,
  title={On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models},
  author={Wang, Shuoyuan and Luo, Beier and Zeng, Hao and Yu, Chengyao and Zhang, Songxin and Xie, Zejian and Jing, Bingyi and Wei, Hongxin},
  journal={arXiv preprint arXiv:2609.32470},
  year={2026}
}