Reinforcement Learning for LLM Post-Training · Module 24
Calibration-Aware RL: Proper Scoring Rules, the RLCR Reward, and Why Binary Rewards Teach Guessing
A binary verifier pays the same for a derived answer and a lucky guess, and it pays for guessing at any success probability above zero. This module derives why a bounded strictly proper scoring rule, the Brier term in RLCR, repairs that incentive; reads RLCR, Behaviorally Calibrated RL and Kalai et al. against their own tables; and ends with what a Brier term in your own RLVR run does and does not buy.
1. Why a binary verdict teaches guessing
1.1 The incentive, in one line
Take one prompt and one candidate answer \(y\) whose probability of being judged correct is \(p_y\). Under the reward \(R=\mathbf 1[y=y^*]\) the expected reward of answering is \(p_y\), and the expected reward of saying "I don't know" is 0, because a verifier has no category for abstention and scores it as wrong. Answering wins whenever \(p_y>0\). Kalai et al. state the general form as Observation 1: for any belief over binary graders, an abstention is never among the optimal responses2 (§4.1). Wu et al. draw the consequence for RLVR: binary rewards make models "good test-takers" and encourage "guessing whenever the probability of correctness exceeds zero"3 (abstract). Damani et al. add the second half of the diagnosis: such a reward pays "equally whether they are confidently correct or merely guessing" and penalizes "identically whether they abstain or produce incorrect answers"1 (§1).
R1-Zero is the clean instance: its reward "is solely based on the correctness of final predictions against ground-truth answers"4 (§1), plus a format term (Module 10, Section 2). Neither term reads the model's stated certainty, which has two consequences.
- Guessing is free. A rollout that guesses and lands on the right answer earns the same as one that derived it. Module 10 (Section 5.4) shows how guessable multiple-choice prompts feed this, and Module 20 (Section 8.3) shows how large-\(k\) pass@k counts lucky guesses as capability.
- Confidence is unconstrained. If the policy writes "confidence: 95%" or hedges, \(\partial R/\partial q = 0\): the reward does not change. A token the reward does not read is shaped only by what happens to correlate with reward in the sampled data and by what the base model brought with it.
1.2 What post-training did to calibration, as measured
The GPT-4 technical report is the earliest widely cited measurement: the pre-trained model "is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct)", and "after the post-training process, the calibration is reduced"5 (§5, Fig. 8, an MMLU subset scored on the probabilities of the A/B/C/D tokens). That is token-probability calibration after PPO-based RLHF. For the verbalized kind, Leng et al. report that RLHF "tends to lead models to express verbalized overconfidence" and trace it to PPO reward models with "inherent biases towards high-confidence scores regardless of the actual quality of responses"6 (abstract). For reasoning RL, AbstentionBench finds that "reasoning fine-tuning degrades abstention (by 24% on average), even for math and science domains on which reasoning models are explicitly trained", across 20 datasets7 (abstract-level reading).
The most direct evidence sits in the RLCR appendix. Its RLVR baseline is trained on binary correctness and then prompted at evaluation time to state a confidence. A reporter that always states a confidence of 1 has Brier score equal to the error rate and ECE equal to \(1-\text{accuracy}\), and an AUROC of exactly 0.5, because every confidence ties. Table 1 compares those predictions with what the paper prints.
| Dataset | RLVR accuracy (%) | 1 − accuracy | Printed Brier | Printed ECE | Printed AUROC |
|---|---|---|---|---|---|
| HotpotQA (in-domain) | 63.0 | 0.370 | 0.37 | 0.37 | 0.50 ± 0.00 |
| TriviaQA | 62.2 | 0.378 | 0.38 | 0.38 | 0.50 ± 0.00 |
| SimpleQA | 12.4 | 0.876 | 0.88 | 0.88 | 0.50 ± 0.00 |
| CommonsenseQA | 90.8 | 0.092 | 0.09 | 0.09 | 0.50 ± 0.00 |
| GPQA | 39.5 | 0.605 | 0.60 | 0.60 | 0.50 ± 0.00 |
| GSM8K | 80.1 | 0.199 | 0.20 | 0.20 | 0.50 ± 0.00 |
| MATH-500 | 38.6 | 0.614 | 0.61 | 0.61 | 0.50 ± 0.01 |
All seven match to two decimals. The reading, which is mine and not the paper's: the RLVR model's stated confidence carries no information and sits at or very near 1. That is the failure the Brier term is meant to repair, and it also means the headline "ECE 0.37 to 0.03" compares a model trained to emit a confidence against one that was only asked to. The informative baselines are the classifiers and probes of Section 5.
1.3 Three things called calibration
The literature uses one word for three quantities, and they fail and are repaired separately. Table 2 separates them; the rest of the module is about the second and third.
| Notion | What is read | Set by | Evidence |
|---|---|---|---|
| Token-probability | Probability of the answer tokens | Pretraining cross-entropy; degraded by post-training | Base GPT-4 calibrated, post-trained not52. RLCR's "Answer Probability" baseline has ECE 0.36 on HotpotQA because "the model typically commits to an answer during CoT, inflating output confidence"1 (§4.2) |
| Verbalized | A number the model writes | RL reward on that number (RLCR, Wu et al.) | Sections 3 to 6 |
| Behavioral | Whether it answers or abstains as a function of a stated threshold | Grading with explicit targets; thresholding a verbalized \(q\) | Sections 6 and 7 |
For an inference engineer the first row has a practical edge: log-probabilities of the final answer tokens are free at serving time. Table 7 shows what they are worth after RL. As a ranking they are mixed, with AUROC 0.72 on HotpotQA, the highest in that table, but 0.52 on math1 (Table 1). As probabilities they are inflated (ECE 0.36 on HotpotQA), because the answer is nearly deterministic once the chain of thought has committed to it. A raw log-probability threshold needs recalibration on held-out data before it means anything, and entropy collapse during RL (Module 13) plausibly makes that worse (my inference).
2. Proper scoring rules, and what the Brier term pays for
2.1 Definition and the Brier derivation
A scoring rule for a binary event assigns a loss \(\ell(q,c)\) to a stated probability \(q\in[0,1]\) and a realized outcome \(c\in\{0,1\}\). It is proper when the expected loss under \(c\sim\mathrm{Bernoulli}(p)\) is minimized at \(q=p\), and strictly proper when that minimizer is unique81 (Gneiting and Raftery, abstract and §2; Damani et al., §2). Gneiting and Raftery write scores as rewards to be maximized; the RLCR proof writes losses. This module writes losses, so a reward is \(\lambda c-\ell(q,c)\).
The Brier loss is \(\ell=(q-c)^2\). Its expectation is
$$p(1-q)^2+(1-p)\,q^2=p(1-p)+(q-p)^2,$$an irreducible term plus the squared miscalibration, so it is uniquely minimized at \(q=p\). The RLCR reward is \(R=c-(q-c)^2\)1 (Eq. 1). Because \(c^2=c\) for a 0/1 outcome, \(R=2qc-q^2\): it is \(2q-q^2\) when the answer is correct and \(-q^2\) when it is wrong. Taking the expectation over \(c\) gives
$$\mathbb E[R\mid p,q]=2pq-q^2=p^2-(q-p)^2 .$$The expected reward is a downward parabola in the stated confidence. It peaks at \(q=p\), where it equals \(p^2\), and it falls by exactly the squared miscalibration. Two reference points are worth memorizing. Stating \(q=0\) pays 0 whatever happens (correct: \(1-1\); wrong: \(0-0\)). Stating \(q=1\) pays \(+1\) or \(-1\), which is \(2p-1\) in expectation.
2.2 Which rules qualify: a mixture of thresholds
Gneiting and Raftery's Theorem 3 (due to Schervish) says every regular, left-continuous proper rule for a binary event is a mixture, with a nonnegative measure \(\nu\) over cost-loss ratios (thresholds) \(t\in(0,1)\), of asymmetric zero-one scores8 (§3.2, Eqs. 14 and 15). Each component grades a single decision, "act if the forecast exceeds \(t\)", not the number itself. The rule is strictly proper if and only if \(\nu\) puts positive mass on every open interval. Table 3 lists the three rules RLCR names.
| Rule | Loss \(\ell(q,c)\) | Gap \(\ell(p,1)-\ell(p,0)\) | Density over thresholds \(t\) | Bounded gap? |
|---|---|---|---|---|
| Brier | \((q-c)^2\) | \(1-2p\), at most 1 | uniform | yes |
| Spherical | \(-\dfrac{qc+(1-q)(1-c)}{\sqrt{q^2+(1-q)^2}}\) | \((1-2p)/\sqrt{1-2p+2p^2}\), at most 1 | \((1-2t+2t^2)^{-3/2}\) | yes |
| Logarithmic | \(-c\ln q-(1-c)\ln(1-q)\) | \(\ln\frac{1-p}{p}\), unbounded as \(p\to0\) | \((t(1-t))^{-1}\), an infinite measure | no |
Brier weights every threshold equally. The log score piles up weight near 0 and 1 without limit, which is the same fact as its unboundedness; Gneiting and Raftery note that it "has been criticized for its unboundedness"8 (Example 3). Section 3.2 shows why the bound matters for training, and Section 6 shows that Wu et al. rebuild both the Brier and a clipped log-style reward by choosing the measure \(\nu\).
2.3 What the term pays for
Calibration alone is cheap. Suppose a policy's prompts have success probabilities \(p_i\) with mean \(\bar p\). It can state \(q=\bar p\) on every prompt, be calibrated on average, and earn \(\mathbb E[2p\bar p-\bar p^2]=\bar p^2\). A reporter that knows each \(p_i\) earns \(\mathbb E[p^2]=\bar p^2+\mathrm{Var}(p)\). The reward therefore pays exactly \(\mathrm{Var}(p)\) for knowing which prompts the policy will get right, at most \(\bar p(1-\bar p)\) when every \(p_i\) is 0 or 1 (derived here). ECE can be driven to zero by a constant; the Brier reward is the term that pays for telling easy prompts from hard ones, and Section 5 uses that yardstick.
| \(p\) | Binary reward, answer | RLCR, honest \(q=p\) | RLCR, \(q=1\) | RLCR, \(q=0\) | Threshold \(t=0.9\), answer |
|---|---|---|---|---|---|
| 0.05 | 0.05 | 0.0025 | −0.90 | 0 | −8.50 |
| 0.20 | 0.20 | 0.04 | −0.60 | 0 | −7.00 |
| 0.50 | 0.50 | 0.25 | 0.00 | 0 | −4.00 |
| 0.80 | 0.80 | 0.64 | 0.60 | 0 | −1.00 |
The binary column says "answer" at every \(p\); the \(t=0.9\) column says "abstain" at every \(p\) shown. RLCR gives the policy no abstain action, so the honest-answer column never goes negative: the only way to lose is to state a confidence that is too high. Note that answering at \(q=1\) pays \(2p-1\), which is exactly the threshold grading at \(t=0.5\) (\(p-(1-p)\)): a model that always states 1 is a test-taker graded at one error per hit.
3. The RLCR reward and Theorem 1
3.1 Format, reward, and token order
RLCR trains one model to write four spans in a fixed order: <think> reasoning, <answer>, <analysis> (a written account of what could be wrong) and <confidence> (a number between 0 and 1 inclusive)1 (Fig. 1, App. C). A response with all four tags in order earns the format credit, and "Both format and calibration rewards are weighted equally"1 (App. B.2). Two properties of the layout matter. The confidence is written after the answer is committed, so it is a prediction about text that already exists. And the reward is one scalar per rollout, so the group-relative advantage multiplies every token's gradient, reasoning tokens included (Figure 3; mechanism in Modules 2 and 11). The Brier term reaches the reasoning only through that shared advantage.
The reward itself is short enough to write down. This is a sketch for this module, not code from the paper: the format weight and the handling of an invalid response are my choices, since the sections read give only the equal-weighting sentence above.
import re
PATTERN = re.compile(
r"\s*<think>.*?</think>\s*<answer>(.*?)</answer>\s*"
r"<analysis>.*?</analysis>\s*<confidence>\s*([01](?:\.\d+)?)\s*</confidence>\s*",
re.S,
)
def rlcr_reward(completion, gold, is_correct, fmt_weight=1.0):
"""Eq. 1 of Damani et al. plus a format term. is_correct(answer, gold) is the task verifier."""
m = PATTERN.fullmatch(completion)
if m is None:
return 0.0 # invalid format: no credit, no calibration term (my choice)
q = float(m.group(2))
if not 0.0 <= q <= 1.0:
return 0.0
c = 1.0 if is_correct(m.group(1).strip(), gold) else 0.0
return fmt_weight + c - (q - c) ** 2 # c - (q - c)^2 == 2*q*c - q^2 for c in {0, 1}
3.2 Theorem 1, checked against the paper
The statement, with the paper's expressions elided1 (§3): suppose the success indicator of each prediction \(y\) is Bernoulli\((p_y)\). Then
Calibration incentive. For any y, the expected reward … is maximized when q = py. Correctness incentive. Among all calibrated predictions (y, py), expected reward is maximized by the prediction whose success probability py is greatest.
The first property is propriety: \(\lambda p_y\) does not depend on \(q\), so maximizing the reward over \(q\) is minimizing the expected loss. For the second, let \(W(p)=\lambda p-p\,\ell(p,1)-(1-p)\,\ell(p,0)\) be the best reward attainable at success probability \(p\). Propriety means \(p\,\partial_q\ell(p,1)+(1-p)\,\partial_q\ell(p,0)=0\) at \(q=p\), so by the envelope argument
$$W'(p)=\lambda-\big[\ell(p,1)-\ell(p,0)\big].$$Higher \(p\) never pays less exactly when the gap \(\ell(p,1)-\ell(p,0)\) stays at most \(\lambda\). For the Brier loss the gap is \(1-2p\le1\), \(\lambda=1\) suffices, and \(W(p)=p^2\), which is increasing. For the log loss the gap is \(\ln\frac{1-p}{p}\to\infty\) as \(p\to0\), and no finite \(\lambda\) works. Corollary 1 of the paper says the same: a bounded strictly proper rule has a finite \(\lambda\) that works; for an unbounded one no finite \(\lambda\) is guaranteed to1 (App. A).
Two printing slips are worth knowing before you read Appendix A. Its last displayed line prints \(W'(p)=S(p,1)-S(p,0)\), dropping the leading "\(\lambda-\)"; the line above it and the conclusion after it both carry the \(\lambda\), and the check \(W=p^2\Rightarrow W'=2p=1-(1-2p)\) agrees with the corrected form. And the main text states the bound as \(S(p,1)-S(p,0)<\lambda\) where the appendix has \(\le\lambda\).
3.3 Why not the log score
The log score is the textbook proper rule, and it fails the correctness incentive. With \(\lambda=1\) and an honest report, the expected reward is \(p(1+\ln p)+(1-p)\ln(1-p)\)1 (Eq. 13), which the paper shows as non-monotone in Fig. 5. Setting its derivative \(1+\ln\frac{p}{1-p}\) to zero puts the minimum at \(p=1/(1+e)\approx0.269\), where the value is \(-0.313\); the curve returns to zero at \(p\approx0.648\) (derived here, natural log assumed). Below that point a response that is certainly wrong and states \(q=0\) earns exactly 0 and beats any honest attempt (Figure 4).
The authors show it happens. In a toy task with five arms, up to five observed draws, and permission to answer "invalid" with \(q=0\), a Qwen2.5-7B trained on the log variant "rapidly converges to a degenerate policy" that outputs the invalid arm for nearly every task1 (App. E.2): accuracy 0, with Brier and ECE of 0.00 because a policy that always says "no answer, zero confidence" is perfectly calibrated. The Brier variant reaches 34.4% accuracy at ECE 0.02 on the same task. On HotpotQA the log variant does not collapse, at 59.5% accuracy against 62.1% and ECE 0.07 against 0.03, and the authors believe hacking is "dependent on both the data distribution as well as the model size"1 (App. E.3). The toy is the regime to worry about: many prompts with low \(p\).
3.4 Variants that fail or fall short
| Setting | Accuracy (%) | ECE | What happened |
|---|---|---|---|
| Toy arm task, Brier | 34.4 | 0.02 | Makes real predictions |
| Toy arm task, log | 0 | 0.00 | Outputs the invalid arm at \(q=0\) |
| Log variant | 59.5 | 0.07 | No collapse; OOD ECE 0.21 |
| Brier (RLCR) | 62.1 | 0.03 | OOD ECE 0.21 |
| Brier only, loss on whole sequence | 0 | 0.00 | Empty or trivial answer at \(q=0\) |
| Brier only, loss on analysis and confidence | 62.0 | 0.08 | Started from the RLVR model; OOD ECE 0.25 |
| Ternary abstention reward, \(\lambda=0.5\) | 62.1 | 0.31 | Started from base; OOD ECE 0.35 |
| RLVR | 63.0 | 0.37 | Constant confidence (Table 1 here); OOD ECE 0.46 |
A Brier-only reward is optimized by a response with no content: the authors call the degenerate output "essentially the optimal policy with a calibration-only reward" and avoid it by masking the loss to the analysis and confidence spans1 (App. I; mask mechanics in Module 18). Earlier calibration-only RL used the Brier score (Xu et al.) or a clipped log loss (Stangel et al., "Rewarding Doubt"), both on non-reasoning tasks1 (§5); the latter states a reward "based on the logarithmic scoring rule"9 (abstract-level reading).
The ternary reward (+1 correct, 0 wrong, \(\lambda\) for abstaining; Wei et al., Mohamadi et al. and Song et al. as cited by Damani et al.) has a clean relation to Kalai et al.'s grading. Answering beats abstaining when \(p>\lambda\). Subtracting \(\lambda\) and dividing by \(1-\lambda\) maps the payoffs to \((1,\,-\lambda/(1-\lambda),\,0)\), which is threshold grading at \(t=\lambda\) (derived). The authors' complaint is that it "only teaches whether the model's internal confidence exceeds the threshold \(\lambda\)" and "can also suppress exploration"1 (App. I).
4. How the reward reaches the policy
4.1 A toy group
Take one prompt and eight rollouts, each with a correctness bit and a stated confidence. The numbers are constructed for this module, not taken from a paper. Table 6 gives the binary and RLCR rewards and the group-relative advantage \(A=r-\bar r\), without the division by the group standard deviation that RLCR removes1 (App. B.2; Module 11 covers what that division does).
| Rollout | Correct \(c\) | Stated \(q\) | Binary reward | Binary \(A\) | RLCR reward | RLCR \(A\) |
|---|---|---|---|---|---|---|
| 1 | 1 | 0.9 | 1 | +0.500 | 0.99 | +0.761 |
| 2 | 1 | 0.6 | 1 | +0.500 | 0.84 | +0.611 |
| 3 | 1 | 0.3 | 1 | +0.500 | 0.51 | +0.281 |
| 4 | 1 | 0.6 | 1 | +0.500 | 0.84 | +0.611 |
| 5 | 0 | 0.9 | 0 | −0.500 | −0.81 | −1.039 |
| 6 | 0 | 0.6 | 0 | −0.500 | −0.36 | −0.589 |
| 7 | 0 | 0.3 | 0 | −0.500 | −0.09 | −0.319 |
| 8 | 0 | 0.3 | 0 | −0.500 | −0.09 | −0.319 |
The binary column gives every correct rollout +0.5 and every wrong one −0.5, whatever it said about its confidence. Under RLCR a confident wrong answer (rollout 5) is pushed down 3.3 times as hard as a hedged wrong one (rollouts 7 and 8), and a hedged correct answer (rollout 3, +0.281) is pulled up less than a confident correct one (rollout 1, +0.761). Summing the advantages over the rollouts that wrote each confidence gives \(-0.278\) for 0.9, \(+0.634\) for 0.6 and \(-0.356\) for 0.3, so probability mass moves toward the value nearest the group's success rate; the binary sums are 0, +0.5 and −0.5, which only reflect which confidences happened to co-occur with correct answers. Eight rollouts are a noisy sample, so read this as the mechanism, not a prediction.
The general statement is cleaner. If the stated \(q\) is independent of correctness, as for the constant reporter of Table 1, a binary reward gives it zero expected advantage whatever its value, because \(\mathbb E[c-\bar p\mid q]=0\). The Brier term gives it the gradient \(\partial_q\mathbb E[R]=2(p-q)\), which points at \(p\). Nothing in RLVR moves a policy off a constant; RLCR does.
4.2 The recipe and the data
RLCR uses GRPO with modifications on a Qwen2.5-7B base with no KL term. It removes the standard-deviation division, following Turtel et al., which "might help with learning on examples where there are extreme miscalibrations"; aggregates token losses with the BNPO normalizer; and samples 32 responses per prompt at temperature 0.7 with an effective batch of 2,048, for one epoch, capping completions at 1,536 tokens (HotpotQA) and 4,096 (math)1 (App. B.2). The HotpotQA set is 20,000 examples in equal thirds that keep 0, 1 or 2 of the gold paragraphs among 8 shown, so a third of the training prompts cannot be answered from the context by construction. The math set is 15,000 Big-Math problems with a LLaMA-8B solve rate between 0 and 70% and numeric answers, graded by math-verify1 (App. B.1). The SFT variant first fine-tunes on 500 base-model solutions extended with DeepSeek-R1-written uncertainty analyses1 (App. B.2).
Two design points follow from the reward (my reading). A calibration reward needs prompts that span the probability range; the HotpotQA construction supplies low-\(p\) prompts on purpose, whereas the mid-range pass-rate filters of Module 10 (Section 5) would remove the prompts whose right answer is \(q\) near 0 or 1. And the reward turns verifier errors into calibration targets. The authors found that "verifier noise can be significant in Math datasets and can cause training instability" and restricted the set to numeric answers1 (App. B.1). If the verifier rejects a correct answer with probability \(\delta\), the optimal stated confidence is the pass probability, roughly \(p(1-\delta)\) plus false accepts, so a noisy verifier makes the policy under-confident by construction (derived).
4.3 Zero-variance groups
Under a binary reward a group that is all correct or all wrong has zero advantage, and DAPO-style dynamic sampling drops it (Module 13, Section 3.2). Under the Brier term a group in which every rollout is correct still has reward variance if the stated confidences differ: for \(q=0.9\) and \(q=0.5\) the rewards are 0.99 and 0.75, and the advantages push \(q\) up, the right direction as \(p\to1\). A filter keyed on accuracy variance would discard exactly the groups where confidence is the only thing that varies; key it on reward variance instead. RLCR does not discuss dynamic sampling, so this is derived from the reward definition, not measured.
4.4 The critic already holds a calibrated number
Wu et al. observe that a PPO critic trained with squared error against a 0/1 return estimates the success probability, so it minimizes a Brier-type loss and is calibrated by construction3 (§3.2.3). They read the critic's value at the final token as a confidence ("Critic Value"). On BeyondAIME it gives the lowest smECE in their table, 0.061, with an SNR gain of 1.202 and a confidence AUC of 0.881; transferred from math to SimpleQA, its smECE is 0.665 against 0.459 for the verbalized-Brier variant3 (the paper's Tables 1 and 4). A critic-free recipe (Module 9) gives that readout up, and a Brier term on a verbalized \(q\) puts a comparable number in the policy's own output at the price of tokens (Module 5 covers the critic). For claim-level confidence the critic fails: its value estimates whether the final solution succeeds given the prefix, so it dips at the contradiction, not at the earlier error that caused it3 (§3.3.2).
5. RLCR's results, read against their own tables
5.1 The main table
Table 7 reproduces the rows of Damani et al.'s Table 1 that carry the argument. Every row except Base, RLCR and SFT+RLCR scores the same RLVR model's outputs; the classifier rows add a second model, initialized from Qwen2.5-7B Base and trained on those outputs with binary cross-entropy1 (§4.1). ECE uses 10 bins; AUROC asks whether confidence separates correct from wrong answers.
| Method | Acc. (%) | AUROC | Brier | ECE | OOD acc. (%) | OOD AUROC | OOD Brier | OOD ECE |
|---|---|---|---|---|---|---|---|---|
| Trained on HotpotQA-Modified; in-domain = HotpotQA | ||||||||
| Base | 39.7 | 0.54 | 0.53 | 0.53 | 53.3 | 0.54 | 0.41 | 0.40 |
| RLVR | 63.0 | 0.50 | 0.37 | 0.37 | 53.9 | 0.50 | 0.46 | 0.46 |
| RLVR + BCE classifier | 63.0 | 0.66 | 0.22 | 0.07 | 53.9 | 0.58 | 0.27 | 0.24 |
| RLVR + linear probe | 63.0 | 0.55 | 0.24 | 0.10 | 53.9 | 0.53 | 0.38 | 0.38 |
| Answer probability | 63.0 | 0.72 | 0.36 | 0.36 | 53.9 | 0.60 | 0.42 | 0.42 |
| RLCR | 62.1 | 0.69 | 0.21 | 0.03 | 56.2 | 0.68 | 0.21 | 0.21 |
| Trained on Big-Math digits problems; in-domain = Math (3-dataset average) | ||||||||
| Base | 56.1 | 0.56 | 0.40 | 0.39 | 47.8 | 0.53 | 0.46 | 0.45 |
| RLVR | 72.9 | 0.47 | 0.28 | 0.26 | 52.5 | 0.52 | 0.49 | 0.49 |
| RLVR + BCE classifier | 72.9 | 0.78 | 0.15 | 0.10 | 52.5 | 0.55 | 0.34 | 0.33 |
| RLCR | 72.7 | 0.67 | 0.17 | 0.10 | 50.9 | 0.60 | 0.28 | 0.25 |
| SFT + RLCR | 72.2 | 0.78 | 0.14 | 0.08 | 43.8 | 0.66 | 0.24 | 0.18 |
5.2 What the table supports
Accuracy is matched. In-domain HotpotQA accuracy is 63.0% for RLVR and 62.1% for RLCR, with a 95% bootstrap half-width of ±3.05 on 1,000 validation questions1 (App. K, the paper's Table 11; App. B.3). Math is 72.9% against 72.7%. The calibration term did not cost accuracy at 7B on these tasks.
The classifier is closer than the headline suggests. The BCE classifier reaches ECE 0.07 in-domain against RLCR's 0.03, and 0.24 against 0.21 out of domain, at the price of a second 7B model that the authors call "expensive, as it requires training and inference with two large models"1 (§4.1). On math in-domain they tie at 0.10, and the classifier discriminates better (AUROC 0.78 against 0.67). What RLCR buys over a classifier is one model and one forward pass, not a large calibration gap.
ECE alone is a weak yardstick; use resolution. A constant reporter at the accuracy has ECE near zero. On HotpotQA, RLCR's accuracy of 62.1% gives a constant-reporter Brier score of \(0.621\times0.379=0.235\); RLCR's printed 0.21 is a Brier skill of \(1-0.21/0.235\approx0.11\), between 0.09 and 0.13 given that the Brier score is rounded to two decimals (derived). The stated confidences resolve about a tenth of the variance that a perfect predictor would, and the AUROC of 0.69 tells the same story. By Section 2.3 that is the quantity the reward pays for.
Out of domain the gain is real but the absolute error stays high. Average OOD ECE is 0.21 after HotpotQA training and 0.25 after math training. Per dataset there is no uniform winner: on SimpleQA, where accuracy is 12.1%, RLCR's ECE is 0.34 against 0.06 for the Brier-trained classifier and 0.88 for RLVR; on TriviaQA it is 0.06 against 0.37 for the same classifier1 (App. K, the paper's Table 8). The authors say so: "even after RLCR, out-of-domain calibration error is often high in an absolute sense, and models may still assign high confidence to multiple contradictory answers"1 (§6).
The SFT warmup trades accuracy for calibration. SFT+RLCR has the best math calibration (ECE 0.08 in-domain, 0.18 OOD) but OOD accuracy falls from 50.9% to 43.8%, which the authors call "possibly due to catastrophic forgetting induced by SFT warmup"1 (§4.3).
The effect size depends on the model. On OLMo-2-7B-Instruct, ECE falls from 0.38 to 0.09 in-domain and 0.48 to 0.20 OOD; on Qwen3-8B the drops are smaller, 0.36 to 0.23 and 0.29 to 0.17, at matched accuracy1 (App. F, the paper's Tables 4 and 5).
5.3 What the calibration costs in tokens
| Variant | Acc. (%) | Tokens | ECE | OOD acc. (%) | OOD tokens | OOD ECE |
|---|---|---|---|---|---|---|
| RLVR | 63.0 | 92 | 0.37 | 53.9 | 142 | 0.46 |
| RLVR, analysis prompted | 62.3 | 224 | 0.34 | 52.6 | 221 | 0.39 |
| RLCR, analysis removed at inference | 61.7 | 113 | 0.09 | 56.5 | 179 | 0.26 |
| RLCR | 62.1 | 249 | 0.03 | 56.2 | 300 | 0.21 |
Prompting alone is not the mechanism: asking the RLVR model for an analysis moves ECE only from 0.37 to 0.34, while an RLCR model with the analysis switched off still reaches 0.09. The reward does most of the work, and the written analysis adds the rest. The serving bill is visible in the token column. RLCR generates 249 tokens against 92, a factor of 2.7 or +157 tokens per answer, and 300 against 142 out of domain (2.1 times). Dropping the analysis cuts that to 113 tokens, +21 or +23%, and keeps ECE at 0.09 in-domain but at 0.26 OOD, against 0.46 for RLVR. The relative overhead shrinks when reasoning is long; the paper reports tokens only for these short-answer tasks.
5.4 Confidence at test time, and its limit
Because \(q\) is a probability of success, it can weight votes: confidence-weighted majority voting beats plain majority voting, max-confidence selection, and two likelihood-based baselines. Averaging \(K\) resampled analyses for a fixed answer lowers the Brier score, but the whole axis of the paper's Fig. 3b spans only 0.208 to 0.2141 (§4.4). Wu et al. add the limit. Calibration is a statement across prompts, and selecting among samples of one prompt needs discrimination within it. Their example: two guesses to question I, one right, both stated at 1/2; three guesses to question II, one right, all stated at 1/3. That is perfect calibration with no selection power3 (§4.5). Pass@k and Maj@k protocols, and what they measure, are in Module 20.
6. Behaviorally Calibrated RL: thresholds in, a scoring rule out
Wu et al. (ByteDance Seed with CMU and Fudan; arXiv v3, 28 January 2026) start from the behavioral calibration that Kalai et al. define: a user supplies a risk threshold \(t\), and the model should answer if and only if its confidence is at least \(t\), and otherwise emit an <IDK> token3 (§1, §3.1).
6.1 From a user's threshold to the Brier reward
The per-threshold reward is Kalai's: +1 for a correct answer, 0 for IDK, \(-t/(1-t)\) for a wrong answer (their Eq. 2). The paper's move is to stop conditioning on \(t\). Let the model state a confidence \(q\) and decide afterwards: abstain if and only if \(q<t\). For boundedness they rescale the per-threshold reward to \(+1\) for a correct answer, \(-1\) for a wrong one and \(2t-1\) for abstaining. That is a positive affine map of Kalai's payoffs at each \(t\) (multiply by \(2(1-t)\), add \(2t-1\)), so the best decision at every \(t\) is unchanged. Averaging over a prior \(u\) on thresholds gives
$$R_u(c,q)=\int_0^{q}(2c-1)\,du(t)+\int_{q}^{1}(2t-1)\,du(t).$$For the uniform prior the first integral is \(q(2c-1)\) and the second is \(q-q^2\), so
$$R_u=2qc-q^2=c-(q-c)^2,$$which is the RLCR reward; the paper writes it as \(2p\cdot\text{valid}-p^2\) and notes the decomposition3 (Eq. 3). Figure 6 draws both branches. The paper states that the result is strictly proper when the prior "has positive density anywhere"; the standard condition is positive mass on every open interval (Section 2.2), which the uniform prior and the truncated prior below both meet. A truncated Beta\((0,0)\) prior, with density proportional to \(1/(t(1-t))\) on \((\epsilon,1-\epsilon)\), gives a cross-entropy-style reward on the clipped confidence \(\mathrm{clip}(q,\epsilon,1-\epsilon)\)3 (Eq. 4). That is the log score's measure from Table 3 cut to a finite range, and the cut is what makes it bounded (my reading). The Kalai threshold, the Brier reward and a clipped log reward are one family indexed by the prior over thresholds. Wu et al. credit Damani et al. and Stangel et al. with the proper-scoring-rule reward itself3 (§2); the derivation, the choice of prior, the critic readout and the claim-level variant are theirs.
6.2 Three strategies, and the one that failed
The setup is Qwen3-4B-Instruct-2507 trained on DAPO-Math-17k, with PPO by default and GRPO for the Beta-prior variant "because we discover improved training stability specifically for this reward design"3 (§4.1).
- Explicit risk thresholding samples \(t\sim U(0,1)\) per training instance, states Kalai's instruction in the prompt, and rewards with Eq. 2. It failed in the authors' hands (§3.2.1, Fig. 1, GRPO, evaluated on AIME 2024 over steps 10 to 60): refusal and hallucination rates were insensitive to the \(t\) in the prompt; the policy over-rejected even at \(t=0\), falling below the binary baseline; SNR did not rise with \(t\), giving a near-zero SNR gain; and the model was over-confident at \(t=1\). Their diagnosis is that "explicitly sampling t creates a noisy optimization landscape"3.
- Verbalized confidence is the Section 6.1 reward with the threshold applied after the fact. Uniform prior gives "Confidence-Brier", the Beta prior gives "Confidence-CE".
- Critic value reads the PPO critic (Section 4.4).
6.3 Results, and how to read them
| Model | SNR gain | Conf AUC | Abs Acc | smECE | Brier | NLL | Pred Acc |
|---|---|---|---|---|---|---|---|
| o4-mini | 0.515 | 0.738 | 0.698 | 0.164 | 0.241 | 0.845 | 0.527 |
| GPT-5 | 0.207 | 0.706 | 0.718 | 0.160 | 0.208 | 0.719 | 0.680 |
| Gemini-2.5-Pro | 0.071 | 0.699 | 0.553 | 0.431 | 0.387 | 1.443 | 0.544 |
| Qwen3-4B-Instruct | 0.084 | 0.715 | 0.338 | 0.632 | 0.599 | 2.316 | 0.291 |
| Qwen3-4B + PPO | 0.101 | 0.786 | 0.388 | 0.560 | 0.518 | 1.873 | 0.361 |
| 4B, confidence-min | 0.683 | 0.863 | 0.703 | 0.220 | 0.194 | 0.606 | 0.384 |
| 4B, confidence-Brier | 0.802 | 0.902 | 0.782 | 0.177 | 0.161 | 0.527 | 0.395 |
| 4B, confidence-prod | 0.806 | 0.883 | 0.766 | 0.166 | 0.166 | 0.516 | 0.385 |
| 4B, confidence-CE | 1.097 | 0.854 | 0.771 | 0.174 | 0.179 | 0.722 | 0.364 |
| 4B, PPO-value | 1.202 | 0.881 | 0.797 | 0.061 | 0.141 | 0.436 | 0.387 |
Which number is the headline. The abstract's "0.806 against GPT-5's 0.207" is the confidence-prod row of Table 9. The Brier variant is 0.802, and two variants are higher: CE at 1.097 and the PPO critic at 1.202. The abstract calls the metric the Accuracy-to-Hallucination Ratio gain; the body calls it SNR gain, and the log base is not stated.
What it measures. SNR gain is relative to each model's own answer-everything ratio, and Conf AUC is insensitive to accuracy by design3 (§4.2, §5). On level metrics the picture is mixed. GPT-5 answers 0.680 of the problems correctly against 0.364 to 0.395 for the trained variants; its smECE of 0.160 is no worse than their 0.166 to 0.220 (only the critic reaches 0.061); and they win on Brier (0.141 to 0.194 against 0.208). The defensible reading is about discrimination: confidence separates right from wrong far better after training (AUC 0.854 to 0.902 against GPT-5's 0.706, Figure 7), a gain of about 0.14 to 0.19 over the untrained model's 0.715.
How much to trust the decimals. BeyondAIME is 100 problems3 (§4.2). At an accuracy of 0.39 the standard error of one accuracy estimate is \(\sqrt{0.39\times0.61/100}=0.049\) (derived). The tables print no intervals, and the text read reports no seed counts. Differences in the second decimal between the 4B variants are within noise; the gap between the variants and the untrained model is not. This is the authors' own evaluation and has not, in the sources read, been independently replicated. Module 20 gives the protocol for judging gains of this size.
Transfer. Trained on math only, the Brier variant on SimpleQA has smECE 0.459 against 0.821 for the untrained model, SNR gain 0.411 against GPT-5's 0.498, and a predictive accuracy of only 0.0583 (the paper's Table 4). The abstract says its calibration error is "on par with" Grok-4 and Gemini-2.5-Pro; the table supports that for Gemini-2.5-Pro (0.451) but not Grok-4 (0.307). At 5.8% accuracy a constant reporter would be nearly calibrated, so this is a weak test on the calibration axis.
Claims inside a response. The same machinery flags individual claims. Only the final outcome is labelled, so claim confidences are aggregated into a response confidence, by product or by minimum, and trained with the Brier reward. Product assumes independent claims, which the authors found violated; a 10-step wrong response at a uniform 80% per claim gives \(0.8^{10}=0.107\), matching the paper's "11%"3 (§3.3.1). Minimum aggregation wins at claim level (SNR gain 0.301 against 0.183, AUC 0.788 against 0.741) and product wins at response level3 (the paper's Tables 1 and 3).
7. Kalai et al.: the incentive sits in the grading
7.1 The argument and the audit
Kalai et al. (OpenAI and Georgia Tech, 4 September 2025) argue that language models hallucinate "because the training and evaluation procedures reward guessing over acknowledging uncertainty"2 (abstract). The pretraining half of the paper, a reduction from generation to a binary "is-it-valid" classification problem, is outside this module. The post-training half is the incentive of Section 1 made formal: under binary grading "abstaining is strictly sub-optimal", because an "I don't know" earns the minimum and an overconfident best guess is optimal2 (§4.1). Their illustration is two models that differ only in behavior: Model A signals uncertainty and never hallucinates, Model B always guesses, and B outscores A under 0-1 grading (§1.2).
The audit is Table 2 of the paper. Of ten widely used evaluations (GPQA, MMLU-Pro, IFEval, Omni-MATH, WildBench, BBH, MATH level 5, MuSR, SWE-bench, HLE) nine are marked binary with no credit for an IDK. The tenth, WildBench, gives partial credit, but its 1-to-10 rubric may place an IDK at 3 to 4, below a "fair" response with factual errors at 5 to 62 (App. F). Their conclusion is that a few added hallucination evaluations "won't suffice" and "the numerous primary evaluations must be adjusted"2 (§1.2); Module 20 (Section 8.3) is the evaluation-side twin.
7.2 Confidence targets, and the penalty that does not follow
The proposed change is a sentence in the prompt of every question: "Answer only if you are > t confident, since mistakes are penalized t/(1 − t) points, while correct answers receive 1 point, and an answer of 'I don't know' receives 0 points"2 (§4.2). The expected score of answering is \(p-(1-p)\,\tfrac{t}{1-t}\), which is positive exactly when \(p(1-t)>(1-p)\,t\), that is when \(p>t\). A threshold of \(t=0\) is binary grading. Table 10 sets the printed values beside the formula.
| Target \(t\) | Penalty as printed | Penalty \(t/(1-t)\) | Break-even \(p\) | Status |
|---|---|---|---|---|
| 0 | not given | 0 | 0 | binary grading |
| 0.50 | 1 | 1 | 0.50 | consistent |
| 0.75 | 2 | 3 | 0.75 | printed value does not follow |
| 0.90 | 9 | 9 | 0.90 | consistent |
| 0.667 | not given | 2 | 0.667 | the \(t\) that a penalty of 2 corresponds to |
The printed "\(t=0.75\) (penalty 2)" contradicts the paper's own formula, which gives \(0.75/0.25=3\); a penalty of 2 belongs to \(t=2/3\). The slip changes decisions. At \(p=0.70\), answering is worth \(0.70-0.30\times3=-0.20\) under the formula, so abstain, and \(0.70-0.30\times2=+0.10\) under the printed value, so answer. The two disagree for every \(p\) between \(2/3\) and \(3/4\). arXiv lists only version 1 of the paper as of 2026-10-27, so the slip is in the only text. Wu et al. use the formula, not the printed value3 (Eq. 2).
A second sentence in the same section inverts its own rule. It says the one behavior optimal for all targets is "outputting IDK among examples where its correctness probability is greater than the target"2 (§4.2), a few lines after stating that answering beats IDK exactly when the probability exceeds \(t\). Read "greater" as "less".
7.3 Behavioral calibration, and how it meets the Brier term
Kalai et al. call the policy that is optimal at every target "behavioral calibration" and define it without a stated probability: "rather than requiring the model to output a probabilistic confidence", the model "must formulate the most useful response in which it is at least t confident", and the property "can be audited by comparing accuracy and error rates across thresholds"2 (§4.2). The Brier reward reaches the same behavior by a different road. A model that states a calibrated \(q\) lets the deployer pick any \(t\) afterwards (Section 6.1), and an answer at \(q=1\) is the \(t=0.5\) test-taker (Table 4). The two differ in where they act. Kalai et al. change the score of an evaluation and run no training experiment on it; Wu et al. tried to train the prompt-conditioned version and moved the threshold out of the prompt when it failed (Section 6.2).
OpenAI's announcement page for the paper returned HTTP 403 when fetched for this module, and I did not work around it, so nothing from it is used except one figure that comes from a search-result summary of the page: gpt-5-thinking-mini abstains on 52% of SimpleQA questions and is wrong on 26%, against 1% abstention and 75% errors for o4-mini. If it holds, it shows the size of the effect the paper describes: a leaderboard that reports only accuracy cannot see that difference.
8. The levers side by side, and what shipped recipes say
| Lever | What changes | Incentive it creates | Left open | Evidence |
|---|---|---|---|---|
| Binary verifier only | nothing | answer at any \(p>0\); \(q\) unconstrained | guessing; constant confidence | Table 1 |
| Brier term on stated \(q\) (RLCR) | reward | \(q=p\); answer with the highest \(p\) | no abstain action; OOD ECE 0.21 to 0.25; +157 tokens | Tables 5, 7, 8 |
| Prompted threshold, trained in | prompt and reward | answer iff \(p>t\) | policy did not become \(t\)-sensitive in the 4B run | Section 6.2 |
| Ternary abstention reward | reward | abstain iff \(p<\lambda\) | no confidence number; fewer attempts | Table 5 |
| Critic or classifier readout | a second head or model | success-probability estimate | extra model; weaker transfer; no critic in GRPO | Tables 7, 9 |
| Refusal SFT data | data | refuse where samples are consistently wrong | not RL; judge-dependent | below |
| Confidence targets in grading | evaluation | explicit threshold \(t\) | needs adoption; no training result | Section 7 |
A keyword search made for this module (calibrat*, abstain, abstention, hallucinat*) of twelve recipe reports in this course, from Llama 3 to DeepSeek-V4, found no calibration or abstention reward. Absence from a report is not absence from practice; the hits are informative.
- Kimi K2 writes prescriptive rubrics that forbid hedging such as "I might be wrong", so the framework "may favor responses that appear confident and assertive" and the model "may occasionally overstate certainty"; the report adds that future iterations "may incorporate more fine-grained handling of calibrated uncertainty"10 (App. F.3). That is the incentive of Section 1 produced by a judge instead of a verifier (Modules 8 and 16).
- Qwen3 tracks an in-house CounterFactQA, questions with false premises that the model should flag: 50.4 after reasoning RL, 61.3 after thinking-mode fusion and 68.1 after general RL, in thinking mode11 (Table 22). Its general RL includes RAG rewards meant to reduce hallucination (§4.4). The report isolates no cause, and flagging a false premise is a cousin of abstention, not calibration.
- Llama 3 takes the data route. It samples answers to questions generated from pre-training snippets, judges them against the snippet, and writes a refusal when responses are "consistently informative and incorrect", on the principle that post-training should align the model to "know what it knows"12 (§4.3.6). There is no RL and no confidence number.
9. Adding a Brier term to your run: what it does and does not buy
It incentivizes
- \(q\to p\) on every answer the policy emits; the expected reward is \(p^2-(q-p)^2\) (Section 2.1).
- Among calibrated outputs, the answer with the highest \(p\), provided the rule is bounded (Theorem 1).
- Resolution: reward grows by \(\mathrm{Var}(p)\) as confidence separates easy prompts from hard ones (Section 2.3).
It does not incentivize
- Abstaining. No action pays for silence: \(q=0\) pays 0 and an honest answer pays \(p^2\ge0\), so the abstain decision is the deployer's threshold on \(q\) (derived).
- Calibration off the training distribution: OOD ECE is 0.21 and 0.25 (Table 7).
- Choosing among samples of one prompt (Section 5.4).
- Faithful uncertainty reasoning. At 7B a classifier reads the confidence off the solution itself; the written analysis helped only the 0.5B and 1.5B classifiers1 (App. G).
- Robustness to a noisy verifier (Section 4.2) or cheap serving (Section 5.3).
Where experts disagree
Everyone in this module agrees on the diagnosis: a binary verdict pays for guessing. They disagree on where to repair it. One camp changes the reward the policy optimizes, one changes the score the field reports, and a third leaves the policy alone and reads confidence from a second component.
Train calibration into the policy with a bounded proper score
Damani et al., MIT, ICLR 20261; Wu et al., ByteDance Seed3
Put a Brier-type term on a verbalized confidence next to the correctness reward. Theorem 1 says the optimum is a calibrated, accurate answer, and the experiments say accuracy survives.
- Accuracy matched at 7B (63.0% against 62.1%, within ±3.05); ECE 0.37 to 0.03 and 0.26 to 0.10.
- One model, one decode; out-of-domain ECE is lower than RLVR's in both runs.
- The same reward falls out of averaging Kalai's thresholds, so it also serves a deployer who picks \(t\) later.
- +157 tokens per answer in the short-answer setting; OOD ECE stays at 0.21 to 0.25.
- Brier skill against a constant reporter is about 0.11; the evidence is 4B to 8B models on short, verifiable tasks, evaluated by the authors.
- A log-score variant collapsed in the toy task; a noisy verifier becomes a biased target.
Fix how benchmarks grade
Kalai et al., OpenAI and Georgia Tech2
Models guess because leaderboards reward guessing; nine of ten audited evaluations give an IDK nothing. Put the target \(t\) and the penalty \(t/(1-t)\) in every prompt.
- Acts on the cause for every lab at once, at no training cost.
- The target behavior is auditable without any stated probability.
- No training experiment in the paper, and its adoption depends on leaderboard owners.
- Wu et al.'s prompt-conditioned training did not become sensitive to \(t\) at 4B3.
- Two printed slips in the key section (penalty 2 at \(t=0.75\); "greater" for "less").
Keep the reward binary; read confidence from a separate head
The classifier and probe baselines of Damani et al.1; the PPO critic readout of Wu et al.3
A second model trained on the policy's outputs, or the critic that PPO already has, gives a success-probability estimate without touching the policy's reward.
- No risk to the policy's accuracy; BCE classifier ECE is 0.07 in-domain against RLCR's 0.03.
- The critic had the lowest smECE in Wu et al.'s table (0.061); retrofits to a finished model.
- A second 7B model, or a critic that GRPO does not have.
- The policy itself still never abstains; transfer is uneven (critic smECE 0.665 on SimpleQA; Brier-trained classifier ECE 0.37 on TriviaQA).
The first two are complements: they act on different layers of one incentive, and each predicts that a binary verdict pushes toward guessing. If you train, the evidence supports a bounded Brier term next to a verifiable reward at modest scale: it costs little accuracy and removes the constant-confidence failure. It does not give you an abstain action, calibration that survives a distribution shift, or a selector within a prompt. If you publish numbers, the grading argument applies whatever you train, and reporting accuracy and error at several targets costs nothing. The third position fits only when you cannot change the RL recipe. Three results would settle the open parts: a matched comparison at frontier scale on agentic or noisy-verifier tasks; independent replication of Wu et al.'s numbers on more than 100 problems, with intervals; and a test of whether thresholding RLCR's \(q\) at \(t=0.5,0.75,0.9\) reproduces the accuracy-and-error trade that Kalai et al. want graders to measure.
What is changing
As of 2026-10-27 three papers are the public evidence for training calibration into reasoning RL: RLCR (v2 15 May 2026, ICLR 2026), Wu et al. (v3 28 January 2026) and Kalai et al. (v1 only). No recipe report in the Section 8 search describes a calibration reward, and Kimi K2 names the gap. Open: full distributions over answers, which RLCR calls impractical for chain-of-thought models1 (§3, footnote 1); long-form and agentic outputs, where the correctness label is a judge's; and whether leaderboards adopt confidence targets. The acronym collision with RLCD, and the claims made under that name, are Module 8 (Section 8) and Module 25; nothing here describes any company's method.
References
- paperDamani, M., Puri, I., Slocum, S., Shenfeld, I., Choshen, L., Kim, Y., Andreas, J. Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (RLCR, Reinforcement Learning with Calibration Rewards). ICLR 2026; arXiv:2507.16806 v2, 15 May 2026 (v1 22 Jul 2025). https://arxiv.org/pdf/2507.16806v2. Accessed 2026-10-27. Used for: Eq. 1 and Theorem 1 (§3), proof and Corollary 1 (App. A), training details and data (App. B), prompts (App. C), log-versus-Brier analysis and toy task (App. E), other models (App. F), classifier-size ablation (App. G), RL-calibration baselines (App. I), per-dataset tables with bootstrap intervals (App. K), Tables 1 to 3.
- paperKalai, A. T., Nachum, O., Vempala, S. S., Zhang, E. Why Language Models Hallucinate. OpenAI and Georgia Tech; arXiv:2509.04664 v1, 4 Sep 2025 (the only version listed on 2026-10-27). https://arxiv.org/pdf/2509.04664v1. Accessed 2026-10-27. Used for: Observation 1 and the post-training argument (§1.2, §4.1), confidence targets and behavioral calibration (§4.2), benchmark audit (Table 2, App. F); the t = 0.75 penalty and the "greater than" slips are in §4.2.
- paperWu, J., Liu, J., Zeng, Z., Zhan, T., Cai, T., Huang, W. Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning. ByteDance Seed, Carnegie Mellon, Fudan; arXiv:2512.19920 v3, 28 Jan 2026. https://arxiv.org/abs/2512.19920. Accessed 2026-10-27. Used for: full text read for this module (the source map had the abstract only): threshold-averaging derivation and the three strategies (§3), explicit-thresholding failure (§3.2.1, Fig. 1), training setup (§4.1), Tables 1 to 4, test-time scaling limit (§4.5).
- paperDeepSeek-AI (Guo, D., et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 v2, 4 Jan 2026 (expanded Nature version). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-27. Used for: R1-Zero reward "solely based on the correctness of final predictions" (§1).
- paperOpenAI. GPT-4 Technical Report. arXiv:2303.08774 v6, 4 Mar 2024. https://arxiv.org/pdf/2303.08774. Accessed 2026-10-27. Used for: pre-trained model calibrated, post-training reduces calibration (§5 Limitations, Fig. 8).
- paperLeng, J., Huang, C., Zhu, B., Huang, J. Taming Overconfidence in LLMs: Reward Calibration in RLHF. arXiv:2410.09724 v2, 28 Feb 2025. https://arxiv.org/abs/2410.09724. Accessed 2026-10-27. Used for: RLHF verbalized overconfidence and reward-model bias toward high-confidence scores (abstract only; the full text was not read).
- paperKirichenko, P., Ibrahim, M., Chaudhuri, K., Bell, S. J. AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions. arXiv:2506.09038, 10 Jun 2025. https://arxiv.org/abs/2506.09038. Accessed 2026-10-27. Used for: reasoning fine-tuning degrades abstention by 24% on average, 20 datasets and 20 models (abstract only).
- paperGneiting, T., Raftery, A. E. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association 102(477), March 2007, pp. 359-378. https://sites.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf. Accessed 2026-10-27. Used for: definitions of proper and strictly proper rules, Savage and Schervish representations (§3.1-3.2, Theorem 3), Table 1 mixing measures, remark on the log score's unboundedness (Example 3).
- paperBani-Harouni, D., Pellegrini, C., Stangel, P., Özsoy, E., Zaripova, K., Navab, N., Keicher, M. Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models. arXiv:2503.02623 v6, 28 Feb 2026 (v1 4 Mar 2025; cited by Damani et al. and Wu et al. as Stangel et al.). https://arxiv.org/abs/2503.02623. Accessed 2026-10-27. Used for: reward based on the logarithmic scoring rule, penalizing over- and under-confidence (abstract only).
- paperKimi Team (Moonshot AI). Kimi K2: Open Agentic Intelligence. arXiv:2507.20534 v2, 3 Feb 2026. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-27. Used for: rubric limitations on hedging and overstated certainty (App. F.3).
- paperQwen Team. Qwen3 Technical Report. arXiv:2505.09388 v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-27. Used for: CounterFactQA by stage (Table 22, §4.7) and RAG rewards in general RL (§4.4).
- paperLlama Team, AI @ Meta. The Llama 3 Herd of Models. arXiv:2407.21783 v3, 23 Nov 2024. https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-27. Used for: factuality post-training by knowledge probing and refusal data (§4.3.6).