Reinforcement Learning for LLM Post-Training · Module 16
Rewards Beyond Ground Truth: LLM Judges, Rubric Rewards, Generative Reward Models, and Judges Trained Inside the Loop
Module 10 covered rewards a program can check. This module covers the rest: reference-based judges, rubric rewards, Kimi's self-critic and tournament rewards, and generative reward models that DeepSeek and others now train alongside the policy. Afterwards you can pick a reward source for a task with no verifier, estimate what the judge costs per step, and name the exploit that source invites and the guard each report used against it.
1. The reward-source ladder
1.1 Four sources, one routing decision
A reward has to be computed once per sampled completion, at rollout rate, for every prompt in every step (Module 1). When a program can do that from an answer key or tests, Module 10 applies. When it cannot, the reports read for this track fall back on three other sources, ordered here by how much of the reward depends on a model's judgment. Most recipes use several at once; the design question is which task goes to which source.
| Source | What produces the reward | Needs | Reported strength | Reported weakness |
|---|---|---|---|---|
| 1 Verifier | program: match, unit tests, constraint check | answer key or tests | "high precision, preventing issues like reward hacking"2 (§4.4) | cannot grade what has no rule; R1 calls reliable reward models "difficult to construct for certain tasks, such as writing"1 (G.1) |
| 2 Reference-based judge | LLM reads prompt, response and a reference answer | one reference per prompt | "avoiding false negatives that can occur with purely rule-based rewards"2 (§4.4) | still an LLM: false positives on trivial openers (Section 2.3) |
| 3 Rubric-guided generative judge | LLM scores against per-prompt criteria, usually after writing reasoning | a rubric: human, LLM or judge written | "more robust to such exploitation" than outcome reward models3 (§3.4) | "higher variance"3; exploits specific to the rubric (Section 4) |
| 4 Scalar preference model | trained scalar head on pairwise labels | preference pairs (Module 4) | "low-variance signals and high training efficiency"3 (§3.4) | "more susceptible to reward hacking"3; R1's helpful reward model was hacked within the stage1 (B.5) |
Two things follow from how the reports use these. First, a recipe is a router. Qwen3's General RL covers more than 20 tasks, each given one of three reward types: rule-based, model-based with a reference answer (Qwen2.5-72B-Instruct as judge), or model-based without one (a preference model)2 (§4.4). GLM-5 blends rule-based rewards, outcome reward models and generative reward models because "each has distinct strengths and weaknesses, and their combination is key to a stable, efficient, and scalable General RL training process"3 (§3.4). Second, a rung is not a fixed place. In DeepSeek-V4 the rung-3 judge is the policy itself (Section 7), which moves a cost out of the reward pipeline and a risk into it.
1.2 The 2025 baseline: R1's stage-2 mixture
R1 is the clean "before" picture. Its second RL stage sums three terms1 (§3.2.2, Eq. 8-10):
$$R = R_{\text{reasoning}} + R_{\text{general}} + R_{\text{language}},\qquad R_{\text{reasoning}} = R_{\text{rule}},\qquad R_{\text{general}} = R_{\text{reward model}} + R_{\text{format}}.$$The stage runs 1,700 steps, and "general instruction data and preference-based rewards are incorporated exclusively in the final 400 steps", because "more training steps with the model based preference reward signal may lead to reward hacking"1 (§3.2.2). Appendix B.5 shows the symptom: with the helpful reward model, the reward rises while CodeForces performance falls (Figure 6 there). R1's own account of the remedy is a limit, not a method: for tasks without a reliable signal it used "human annotation to create supervised data, and only conduct RL for hundreds of steps"1 (G.1).
The reward model itself was built from judge labels. For the helpful model, DeepSeek-V3 compared pairs in the Arena-Hard prompt format; each pair was queried four times with the two responses randomly assigned to positions A and B, the four judgments were averaged, and only pairs whose score difference exceeded 1 were kept (66,000 pairs). The authors also ensured that chosen and rejected responses "have comparable lengths" to limit length bias. The safety model is pointwise, trained on 106,000 prompts labelled safe or unsafe1 (§3.1, B.2). So R1's rung 4 already sat on rung-3-style judge labels, with position randomisation and length matching as the only hygiene; both devices recur below.
2. What a judge gets wrong before any policy touches it
2.1 Judge formats and what each costs
DeepSeek's generalist-reward paper separates how a judge produces its output (scalar, semi-scalar with text, or purely generative text) from how it scores (pointwise, one score per response; or pairwise, picking a best)4 (§2.1, Eq. 1-3). A pairwise judge is, in their words, "lacking flexibility to accept single or multiple responses as input", and a scalar head "could hardly generate diverse reward signals for the same response", which blocks sampling-based improvement of the reward itself4. For policy training the formats also differ in cost per rollout group of \(G\) samples.
| Format | Reward value | Calls per prompt | Where used | Measured weakness |
|---|---|---|---|---|
| Pointwise Likert, no reference | integer 1-10, normalised to [0, 1] | \(G\) | Direct-Likert baseline5; Olmo 3 open-ended chat judge, score in [0, 1]6 | coarse scale, ties; verbosity (Section 2.2) |
| Pointwise checklist | weighted mean of \(k\) binary checks | \(Gk\) explicit, \(G\) implicit | Rubrics as Rewards5; checklist feedback7 | rubric quality bounds the reward (Section 4) |
| Reference-conditioned verdict or score | correct / incorrect, or a score | \(G\) | R1 SFT-data judge1; Qwen32; Olmo 3 chat-reference6 | "master key" false positives (Section 2.3) |
| Pairwise vs reference | -1, 0, +1 | \(G\) | MiniMax-M18 | reference quality; GenRM length bias |
| Pairwise within group | wins from binary comparisons | \(G-1\) to \(G(G-1)\) | Kimi K2, K3910 | position bias; quadratic cost (Section 9.1) |
2.2 Measured biases and the standard repairs
Zheng et al. quantified the biases that every later recipe designs around11 (§3.3-§3.4). Their position-bias test gave each judge two near-identical GPT-3.5 answers to the same MT-Bench question and asked for a verdict twice, with the order swapped. Under the default prompt, Claude-v1 kept its verdict in 23.8% of cases, GPT-3.5 in 46.2%, GPT-4 in 65.0% (Table 2 there; Claude-v1 picked the first position 75.0% of the time). Their verbosity test padded 23 answers with a rephrased copy of their own numbered list; the judge failed if it preferred the padded answer, which happened for 91.3% of Claude-v1 and GPT-3.5 verdicts and 8.7% of GPT-4's (Table 3 there). Self-enhancement was suggestive but not established: GPT-4 favored itself with a 10% higher win rate and Claude-v1 with 25%, while GPT-3.5 did not, and the authors state the study "cannot determine" whether the effect is real.
The repairs are cheap and appear in the recipes. Swapping positions and declaring a win only if both orders agree is the "conservative" fix. For graded math, a reference-guided judge cut the failure rate from 14 of 20 problems (default prompt) to 3 of 20, against 6 of 20 for a chain-of-thought judge that solves the problem itself11 (§3.4, Table 4). R1's four randomly ordered judgments per pair1, MiniMax-M1's "position-switched consistent judgment"8 (§4.2.1) and J1's consistency reward (Section 7) are the same repair applied to reward training.
2.3 Master keys: a collapsed run with a reference-based judge
Reference-based judging (rung 2) looks safest because the judge only has to compare. Zhao et al. report the opposite in the setting RLVR practitioners use. In an RLVR run with Qwen2.5-72B-Instruct as the reference-based judge, training collapsed: response length fell below 30 tokens while KL to the reference surged, because the policy had learned to emit generic openers such as "Solution" or "Thought process:" that the judge scored as correct12 (§1, Fig. 2). They then measured false positive rates for ten such "master keys" across judges and five benchmarks (Table 1 there). A lone colon made GPT-4o accept up to 35% of responses; "Thought process:" produced false positive rates of 60-90% in LLaMA3-70B-Instruct and Qwen2.5-72B-Instruct on every benchmark; a specialised verifier (General Verifier) accepted 66.8% of MATH responses given a single blank space; GPT-o1 and Claude-4 were also affected12 (§3.3.1).
Their repair is a data fix: 20,000 responses regenerated with chain-of-thought prompting, cut to the first sentence (a reasoning opener with no content), labelled invalid and added to the original 160,000 training examples; the resulting Master-RM judges reach near-zero false positive rates on all ten keys12 (§3.2). Inference-time defenses were mixed: chain-of-thought prompting with five-way majority voting raised false positive rates in their reference-based setting, while omitting the question from the prompt lowered them sharply, a trick the authors say does not carry over to general reasoning tasks12 (§3.4).
2.4 Why optimization raises the stakes
A benchmark attack tries a few strings; a policy queries the judge millions of times and keeps whatever scores. A null model that always emits one crafted, instruction-independent response reached an 86.5% length-controlled win rate on AlpacaEval 2.0, 83.0 on Arena-Hard-Auto and 9.55 on MT-Bench13. For learned scalar reward models, 70-90% of PPO's reward improvement on WebGPT and on the dataset of Yang et al.'s RLCD (Module 8) came from longer outputs, and a length-only reward came close to PPO's win rate (56% vs 58% on WebGPT, 64% vs 63% on that dataset)14 (§3.1, Table 2). For generative judges, MiniMax-M1 found that its GenRMs "preferred longer outputs over potentially superior concise alternatives" and that offline mitigation "frequently failed to prevent length bias during RL training", so it monitors length against task success online and recalibrates the GenRM when the policy lengthens without gains8 (§4.2.2). The exploit catalogue is Module 19; here the point is that every reward source below ships with a guard.
3. Reference-based judges: the answer exists, but no rule can read it
The cheapest step off the verifier rung keeps the ground truth and replaces the string comparison with a model. Qwen3 states the motivation: its second reward type gives Qwen2.5-72B-Instruct a reference answer per query and has it score the response, which "allows for more flexible handling of diverse tasks without requiring strict formatting, avoiding false negatives that can occur with purely rule-based rewards"2 (§4.4). The judge's task shrinks from "is this good?" to "does this match?", which is why Zheng et al.'s reference-guided judge cut grading failures on math from 14 of 20 to 3 of 2011. It does not remove the master-key problem of Section 2.3, because the residual risk is the judge accepting anything that looks like reasoning.
3.1 MiniMax-M1: a judge with a measurable quality metric
MiniMax-M1's general RL set has 25K complex samples in two kinds8 (§4.2.1). For STEM and factual items with ground truth but several valid surface forms, rule-based checkers are inaccurate, so a generative reward model grades consistency between response and ground truth on a five-grade scale. The authors measure the judge two ways: accuracy on a human-annotated benchmark built from response-ground-truth pairs "that rule-based checkers fail to judge accurately", and the gap between Best-of-N selected by the GenRM and pass@N. The second metric is worth borrowing. Pass@N is the oracle; the share of the Best-of-N-over-single-sample gain the judge recovers is a direct measure of how much reward signal survives its errors.
For items with no ground truth (instruction following, creative writing), each query is paired with a reference answer written by internal and external models and screened by an internal quality evaluation. During RL the GenRM compares the policy's response with that reference and returns -1, 0 or 1 (worse, similar, better). Instructions with constraints additionally get a rule-based reward for the constraint. Judge training data uses multiple-blind and position-switched consistent judgments, and a Swiss-round scoring pass with the trained GenRM selects the most suitable reference for each prompt8. Here the reference is another model's answer, so the reward is bounded by the quality and style of that answer; RaR makes the same point about rubrics, whose quality "is impacted by reference quality"5 (§5).
3.2 Olmo 3: a judge that anchors the mix
OlmoRL's chat domain uses two judge rewards: "Chat-reference" compares the response with a provided reference and returns a score in [0, 1]; "Chat-open-ended" returns a score in [0, 1] with no reference. Both run Qwen3 32B with thinking turned off, a 32,768-token input limit and at most 2,048 response tokens6 (§4.4.1). Instruction-following prompts get a rule-based 1 or 0 for "all constraints satisfied". For the Tulu 3 portion of its chat prompts, Olmo 3 extracted reference answers with GPT-4.1 and dropped prompts whose mean F1 against eight sampled responses was below 0.1 or above 0.8, removing noisy and overly difficult items6 (§4.4.2).
The judge's most informative effect is as a counterweight. Training with OlmoRL on IFEval-style prompts alone raised IFEval while lowering AlpacaEval; the final mixed run kept AlpacaEval without hurting IFEval, which the authors attribute to the LM-judge reward ensuring the model "continues to produce well-formed chat responses"6 (§4.5). A rule-checked domain optimized alone over-optimizes toward whatever the rule can see, and a judge-scored domain in the same batch pulls back; this is one run's observation, not an ablation of the judge.
| Report | Judge | Reference | Reward | Guard described |
|---|---|---|---|---|
| Qwen3 General RL2 | Qwen2.5-72B-Instruct | one per query | score (scale not stated) | rule-based rewards kept where possible for precision |
| MiniMax-M18 | own GenRM | ground truth, or model-written answer chosen by Swiss round | five-grade; -1 / 0 / +1 | position-switched judgments; Best-of-N vs pass@N gap; online length monitoring |
| Olmo 3 chat6 | Qwen3 32B, thinking off, 2,048 response tokens | dataset answer, rewritten by GPT-4.1 | score in [0, 1] | F1 prompt filter (0.1 to 0.8); mixed with rule-based domains |
| R1 SFT data1 | DeepSeek-V3 | ground truth | correct / incorrect, with analysis | offline rejection sampling, not an RL reward |
| GLM-5 reasoning RL3 | "domain and source-specific judge models or evaluation systems" | ground truth | binary outcome | not described |
R1's use is the odd one out and the safest: DeepSeek-V3 receives the ground truth and the model's prediction and returns an analysis plus a correct or incorrect label, but the output only selects which rejection-sampled traces enter the roughly 600K-sample SFT set1 (B.3.3, Listing 4). No policy optimizes against it, so the master-key dynamics cannot arise.
4. Rubric rewards: criteria in place of a reference
4.1 The formalisation and what it buys
Rubrics as Rewards (RaR) gives each prompt \(x\) a set of \(k\) criteria \(\{(w_j, c_j)\}\), where \(w_j\) is a weight and \(c_j(x,\hat y)\in\{0,1\}\) is a judge's yes/no verdict on whether response \(\hat y\) satisfies criterion \(j\)5 (§2.1). Explicit aggregation normalises the weighted sum so that prompts with different rubric sizes are comparable; implicit aggregation hands all criteria and their categorical weights to a judge that returns one scalar \(f_\phi\)5 (Eq. 1-2):
$$r_{\text{explicit}}(x,\hat y)=\frac{\sum_{j=1}^{k} w_j\, c_j(x,\hat y)}{\sum_{j=1}^{k} w_j},\qquad r_{\text{implicit}}(x,\hat y)=f_\phi\big(x,\hat y,\{d_j\}_{j=1}^{k}\big).$$RLVR is the special case \(k=1\), \(w_1=1\), with \(c_1\) an exact match or test run5 (Remark 1). The authors generate rubrics of 7 to 20 self-contained items per prompt from a reference answer (using o3-mini and GPT-4o), tag each Essential, Important, Optional or Pitfall, and map the tags to weights 1.0, 0.7, 0.3 and 0.9 in the explicit variant; pitfalls are phrased positively ("avoids misinformation") so satisfying one adds reward5 (§3.2, §4.4). The released RaR-Medicine set has 20,166 prompts with 7.5 rubric items per prompt on average (Table 4 there).
A derived example shows why this helps a group-relative learner. Take the medical example in the paper's appendix, a bicarbonate-dosing question with two Essential items (apply the formula; state the safe dose), three Important (justify partial correction; show the steps; use the patient data accurately), one Optional and one Pitfall5 (Appendix A.1). With the experimental weights the denominator is \(2(1.0)+3(0.7)+0.3+0.9=5.3\). A response that meets every Essential and Important item scores \(4.1/5.3=0.774\); one that states only the safe dose and the formula scores \(2.0/5.3=0.377\). Enumerating which items are met, this seven-item rubric yields 34 distinct reward values, against ten for a 1-10 Likert score. More distinct values mean fewer ties among the \(G\) rollouts of a prompt, so fewer groups collapse to zero advantage (Module 11).
4.2 What the experiments show, and what they cannot
The setup is small: Qwen2.5-7B as the policy, GRPO with 16 rollouts per prompt, a 3,584-token context, temperature 1.0, learning rate \(5\times10^{-6}\), 300 steps on a single node of 8 H100s, and gpt-4o-mini as the judge5 (§4.2, Table 10). The authors report relative gains of up to 31% on HealthBench and 7% on GPQA-Diamond for the best RaR variant over a direct-Likert judge reward5 (abstract). The controlled comparison is the HealthBench ablation, where training uses 3.5K HealthBench prompts that have human-written rubrics and evaluation uses a held-out 1K.
Four readings are supported by the tables. A rubric beats a bare Likert judge by a wide margin (35.9 vs 23.9) but a reference-conditioned Likert judge by a narrower one (35.9 vs 31.7), and rubrics written without the reference land at 32.0, level with it: the reference does as much work as the rubric structure. Human-written rubrics (34.8) and reference-conditioned LLM rubrics (35.9) are comparable, so the reference substitutes for human authorship. A fixed list of generic rubrics underperformed because "generic criteria miss prompt-specific requirements and common failure modes"5 (§5). In a separate ablation richer rubrics beat essential-only ones (37.2 vs 34.9), while weights and pitfall items made minimal difference; the authors suspect synthetic pitfall items are hard to write because they must anticipate the policy's failure modes5 (Table 2).
Rubrics also make small judges usable. Scoring chosen against rejected HealthBench responses, rubric guidance raised pairwise accuracy for every judge size, most for Qwen-7B-Instruct (22.0% to 26.7%), and narrowed the spread across judges from 22.0-25.4% to 25.0-27.9%5 (§5, Table 11).
4.3 Who writes the rubric
| Method | Rubric author and grounding | Size and weights | Training use | Reported result |
|---|---|---|---|---|
| RaR5 | o3-mini or GPT-4o from the reference answer | 7-20 items; Essential 1.0, Important 0.7, Optional 0.3, Pitfall 0.9 | GRPO reward | up to +31% relative on HealthBench vs direct Likert |
| RLCF7 | Qwen2.5-72B-Instruct, from candidate responses of varying quality (0.5B-7B models) to list failure modes | yes/no items, importance 0-100, plus two universal items at total weight 100/100 | pairs for DPO; keep the 40% with the largest criterion gap | +4 hard-satisfaction points on FollowBench, +6 InFoBench, +3 Arena-Hard win rate |
| DR Tulu15 | GPT-4.1, from search results plus contrasting rollouts, updated during training | buffer pruned to the highest-std rubrics | GRPO reward | evolving beats static by up to 2 average points |
| Rubicon16 | humans, LLMs or both; 10,000+ rubrics | vector of dimension scores, non-linear aggregation | staged RL reward | +5.2 points absolute on open-ended benchmarks with 5K+ samples (30B-A3B model) |
RLCF is the clearest case for a rubric against a trained scalar model, and it shows the standard rubric exploit. Optimizing checklist completion produced "responses beginning with long preamble overviews, suggesting reward hacking", so every checklist gets two universal criteria (answer directly; match the context). Against reward-model-labelled DPO on the same prompts, Skywork and ArmoRM labels helped on AlpacaEval and InFoBench but regressed on IFEval, whereas checklist labels helped on all five benchmarks, though Skywork beat them on AlpacaEval "by a large margin"7 (§2, §5.2). The judge is the cost: each item is the mean of 25 samples from Qwen2.5-72B-Instruct, "the computational bottleneck of our pipeline"7 (§3).
4.4 Static versus evolving rubrics
A static rubric is fixed before the policy exists, so it cannot name the failure modes the policy will invent. DR Tulu's RLER keeps two sets per prompt. The persistent set is built before training from retrieved web documents, which gives the rubric writer information the policy lacked at generation time. The evolving set is proposed during training from the current group of rollouts and the existing buffer: positive rubrics capture new, relevant strengths the policy explored; negative rubrics describe undesirable patterns such as copying retrieved text verbatim to inflate citation precision15 (§3.1). To keep the evaluation cost bounded, after every GRPO rollout the buffer removes rubrics whose reward has zero variance across the group and keeps the \(K_{\max}\) with the highest standard deviation. This is the zero-variance filter of Module 11 applied to criteria instead of prompts: a rubric that every rollout satisfies, or none does, carries no gradient.
In ablations starting from an intermediate SFT checkpoint (average over four long-form benchmarks 54.7), 500 steps of RL with generic rubrics gave 54.8, closed-book rubrics 55.6 and search-grounded rubrics 56.7. Removing evolving rubrics cost up to 2 average points, with the gap widening over training, and both rubric variants beat a random-reward control15 (Table 3, Fig. 6). Replacing GPT-4.1-mini and GPT-4.1 with Qwen3-8B, the model DR Tulu starts from, as both judge and rubric writer gave a gain of +4.4 average points over SFT against +5.7 with the GPT judges (58.3 vs 59.6, from an SFT baseline of 53.9, Table 4)15. The final run took roughly 27,000 GPU hours for 4,000 steps15 (Appendix G.2).
4.5 Aggregation beyond a weighted sum
Rubicon argues a weighted sum misses interactions between criteria and lists four alternatives: vetoes (failure on a non-negotiable dimension, such as a reward-hacking detector, zeroes the rest), saturation-aware aggregation (diminishing returns past a threshold), pairwise interaction terms, and non-linear shaping that stretches score differences at the high end16 (§2). It reports a "seesaw effect" when strict constraint following and open-ended creativity are trained jointly, and so trains in stages16 (§3.1). Its defense is a pattern this module returns to: read rollouts with anomalously high reward, name the recurring exploit (here, opening praise of the question and self-evaluation of the response), write it as a rubric, and apply it as a veto in later stages16 (§3.2, Appendix A.1). The paper gives no per-device ablation.
5. Kimi K2's self-critic: a pairwise reward grounded by verifiable rollouts
Module 8 introduced K2's rubrics as a modern form of constitutional principles. Here the question is the reward loop. K2 pairs a "Verifiable Rewards Gym" with a second mechanism for "subjective preferences, such as creative writing and open-ended question answering", in which "the model performs pairwise comparisons to judge its own outputs"9 (§3.2).
5.1 The self-critique rubric reward
The actor generates responses for general prompts, and the K2 critic "ranks all results by performing pairwise evaluations against a combination of rubrics": three core rubrics (clarity and relevance; conversational fluency and engagement; objective and grounded interaction), two prescriptive rubrics "that aim to eliminate reward hacking" (no opening praise of the user's question; no sentences justifying why the response is good), and human-annotated rubrics for specific instruction contexts9 (§3.2.2, Appendix F). Some rubrics can be mandatory, yet K2 "retains the flexibility to weigh them against its internal priors". The critic is bootstrapped, not emergent: the team curated "a mixture of open-source and in-house preference datasets and initialize[d] its critic capability in the SFT stage"9. The report does not say how a ranking becomes a scalar reward or how many comparisons are made per group.
5.2 What the closed loop is for
The stated purpose is grounding: on-policy rollouts from verifiable-reward prompts "are used to continuously update the critic, a crucial step that distills objective performance signals from RLVR directly into its evaluation model", so that "the critic continuously recalibrates its evaluation standards in lockstep with the policy's evolution"9 (§3.2.2). Read mechanically, this fixes two weaknesses of a frozen judge. A frozen scalar model is queried on a policy distribution it was not trained on, the setting in which proxy and gold reward diverge (Module 6). The verifiable prompts supply, at no labelling cost, current-policy samples with known outcomes, a pass and a fail from the same prompt being exactly the pair a critic should rank correctly. That reading is mine: K2 does not give the critic's loss. The load-bearing assumption is that a critic sharpened on math and code transfers to creative writing, and the report offers no ablation of the critic, the closed loop, or the self-critic reward against a frozen alternative; it states the gains qualitatively ("comprehensive performance improvements").
5.3 K2's other judge-scored rewards
The same section uses judges in three more places9 (§3.2.1). Instruction following combines code-interpreter checks for verifiable constraints with LLM-as-judge evaluation for the rest, plus a "hack-check layer" that "specifically detects" cases where the model claims to have satisfied an instruction without complying. Faithfulness uses a sentence-level judge, inspired by FACTS Grounding, that flags factual claims lacking support in context and "serves as a reward model". Safety runs an attack model, a target model and a judge model, with the judge applying a task-specific rubric to emit a binary success or failure label. Each is a rubric-guided judge with a narrow remit; a narrow rubric is easier to audit than a general quality judge, though that is my reading and the report ablates none of them.
K2.5 widens the pattern. Generative reward models are applied "on top of verified reward signals" across chat assistants, coding agents, search agents and artifact-generating agents, and "function not as binary adjudicators, but as fine-grained evaluators" of qualities such as helpfulness, level of detail and aesthetic quality of generated artifacts. To "mitigate reward hacking and overfitting to a single preference signal", K2.5 uses "multiple alternative GRM rubrics tailored to different task contexts"17 (§4.4.2). Rotating rubrics shrinks the surface any one rubric exposes at the cost of a less stationary target; no ablation is given. K2's own limitations note flags what a confident-sounding rubric does to hedging, which Module 8 (Section 7.3) and Module 24 take up.
6. Kimi K3's agentic judge and tournament reward
K3 keeps K2's pairwise logic and changes who writes the rubric. For "non-verifiable general tasks" it uses an Agentic Generative Reward Model, "retaining the tournament-style group reward with binary comparisons as in Kimi K2.5", and requires the judge to follow a mandatory protocol10 (§4.1.2). The K2.5 text read here does not use the word tournament; K2's equivalent is a critic that ranks outputs by pairwise evaluation (Section 5). Policy optimization follows K2.510, whose objective multiplies a clipped ratio by \(r(x,y_j)-\bar r(x)\), the reward minus the group mean, with no division by a standard deviation17 (§4.4.2, Eq. 1).
6.1 The mandatory protocol
The judge must (1) read the outcome, product or text output; (2) generate a rubric; (3) score each candidate against the rubric; and (4) record the rubric-assigned scores in a scorepad10. One difference from K2 follows from the text: the rubric is produced at judgment time rather than drawn from a fixed core-plus-prescriptive list, the design DeepSeek-GRM argues for when it makes principle generation part of reward generation (Section 7). Step 3 says "the rubric" in the singular, which suggests one rubric per judgment so that candidates are scored on a shared scale, but the report does not say whether one rubric serves the whole group, which tools the judge may call, or how the scorepad feeds the comparisons.
6.2 A worked tournament reward
The report describes the aggregation in one phrase. The following is an illustration under the simplest reading, a round-robin in which each candidate's reward is its share of wins, \(r_i=w_i/(G-1)\), followed by the mean-centered advantage of K2.5. All judge outputs and lengths are hypothetical.
In case (a) rewards are evenly spaced, 1, 2/3, 1/3, 0, with mean 0.5 and standard deviation 0.373, so the advantages are \(\pm0.50\) and \(\pm0.17\). Dividing by the group standard deviation, as original GRPO does (Module 9), would rescale them to \(\pm1.34\) and \(\pm0.45\) (population standard deviation; \(\pm1.16\) and \(\pm0.39\) with the \(n-1\) convention). Case (b) shows the guard overriding the judge: it is a constraint, not a penalty term, so no quality margin can buy back a length violation. Whether both candidates over the limit is a tie, a coin flip or a double loss, the report does not say.
6.3 A forced choice always produces spread
With a binary verifier, a group of all passes or all fails has zero variance and zero advantage (Module 11). A forced-choice tournament almost never does. For even \(G\) the total number of wins, \(G(G-1)/2\), cannot be divided equally among \(G\) candidates, so equal rewards are impossible; for odd \(G\) a tie needs a perfectly balanced tournament, such as the three-way cycle at \(G=3\) (2 of the 8 equally likely outcomes). Two candidates the judge cannot tell apart still receive different rewards. I simulated a judge that flips a fair coin on every pair: at \(G=16\) its rewards have a standard deviation of 0.127 against 0.307 for a perfectly transitive judge (4,000 trials, seed 1), so with mean-centered advantages it delivers about 41% of the push of a perfect judge, in random directions. Under per-group standard-deviation normalization both would be rescaled to unit variance and the advantage would no longer show how informative the judge was. This is a property of the aggregation, not a claim about K3's judge, whose accuracy the report does not give; it is why judge consistency on near-ties, where Zheng et al. measured 65% for the best judge they tested (Figure 2), bounds the quality of the signal.
6.4 The verbosity guard among its alternatives
K3 describes the guard as "analogous to the reasoning-effort control": given an initial verbosity \(\ell_0\) "estimated from the cold-start model" and a multiplier \(\sigma\), a candidate whose output exceeds \(\sigma\cdot\ell_0\) "automatically loses the binary comparison". The effort control it mirrors overrides the task reward with \(-1\) for trajectories above \(\tau\cdot b_0(x)\) of a per-problem budget, with \(\tau\) annealed in stages10 (§4.1.2). If \(\ell_0\) is likewise per prompt, the guard anchors length to what the cold-start model would write: it suppresses padding and also caps legitimately longer answers such as research reports. Whether \(\sigma\) is annealed is not stated. The guard is a hard rule inside the comparison, unlike the soft or external guards in other reports, which Section 9.3 compares.
6.5 Gating a judge behind deterministic checks
K3's web-development tasks show the layering pattern. Rewards have two parts: deterministic checks that test application behavior and score structural and pixel-level similarity for replication tasks, and model judging by an internal reward model that inspects source code or interacts with the output artifact. "The reward is zeroed when a project fails to build, runs with errors, or fakes rather than implements the artifact"10 (§4.2.7). The judge's opinion matters only after the checks pass, which removes the cheapest exploits (a nice-looking page that does not run) before the judge can be fooled by them.
7. Judges that are trained, not just prompted
A frozen judge is a fixed target. A trained judge can improve, and judging has a property that makes it trainable with the same RL machinery as the policy: when a preference label or a verifier outcome exists for some pairs, "pick the better response" is itself a verifiable task. The sources split into judges trained separately, judges trained alongside the policy, and one case where the policy is the judge.
7.1 Precursors: judging as a trained skill
Self-Rewarding Language Models score their own candidates with an LLM-as-a-Judge prompt (an additive five-point rubric), pair the best and worst, and train the next model with DPO. On Llama 2 70B, three iterations moved the AlpacaEval 2.0 win rate over GPT-4 Turbo from 9.94% to 15.38% to 20.44%, and the judge's pairwise agreement with held-out human preferences from 65.1% (SFT baseline) to 78.7%, 80.4% and 81.7%. Average generation length grew from 1,092 to 1,552 to 2,552 (units as printed), which the authors note "may be a factor in relative performance"18 (§3, Tables 1 and 4). The loop improves the judge as a by-product, with the length confound every judge-scored reward shares; it is DPO, not online RL.
J1 trains the judge with GRPO on 22K synthetic preference pairs (17K WildChat prompts, 5K MATH), rewarding +1 for a correct verdict and a consistency reward of +1 only when the verdict is correct in both response orders, Zheng et al.'s swap test turned into a training signal19 (§2.1-§2.2). The authors report that J1-Qwen-32B, trained on synthetic data only, beats o1-mini, o3 and the 671B DeepSeek-R1 on some judgment benchmarks19 (abstract). Generative Verifiers train a verifier with the plain next-token objective on Yes/No verdicts with chain-of-thought rationales; in Best-of-N selection this lifted GSM8K from 73% to 93.4%20 (abstract).
7.2 DeepSeek-GRM: principles as part of the output
DeepSeek's Self-Principled Critique Tuning treats the principles a judge applies as something it generates, not a preprocessing step. A pointwise generative reward model is first fine-tuned on rejection-sampled principle-and-critique traces (optionally with a hint of the correct answer), then trained online with GRPO under a simple rule: reward +1 if the scores extracted from its output rank the ground-truth best response uniquely highest, \(-1\) otherwise; no format reward, with a larger KL coefficient to hold the format4 (§3.2, Eq. 11). The motivation: GPT-4o's self-generated principles barely moved Chat Hard (76.1 to 75.9), while principles filtered to those that led to correct rewards raised it to 77.8, so the skill worth training is producing principles that lead to correct judgments4 (Table 1). At inference, summing \(k\) sampled scores on a 1-10 scale expands the reward space \(k\)-fold; a 27B model scored 69.9 overall on reward-model benchmarks with one sample, 71.0 with 32 votes and 72.8 with meta-reward-model-guided voting, against 71.3 reported for GPT-4o4 (Table 2). These are reward-model benchmark scores, not policy-training results. Neither the V3.2 nor the V4 sections read cite this paper, so treat the shared lab as context, not lineage.
7.3 DeepSeek-V4 and the actor as judge
V4 states the design most directly. For hard-to-verify tasks it "dispense[s] with these conventional scalar-based reward models", curates rubric-guided RL data and employs a GRM to evaluate trajectories. "Crucially, we apply RL optimization directly to the GRM itself": the actor "natively functions as the GRM, enabling the joint optimization of the model's evaluative (judging) proficiency alongside its standard generative capabilities". The claimed benefits are that judging is fused with the model's reasoning, "resulting in highly robust scoring", and that it needs "only a minimal set of diverse human annotations"21 (§5.1.1). The section gives no ablation, no judge accuracy and no comparison with a separate judge. Nemotron 3 Ultra takes the other route for chat: a principle-following GenRM built on the Ultra SFT model, trained with the RLVR method NVIDIA used elsewhere to predict a score for each of two responses plus a ranking, whose overall score is the RLHF reward; changing principles across RLHF iterations lets it target weaknesses "without retraining the reward model itself". Its stated motivation is that larger policies exploit weaker reward models, and that reasoning GenRMs reduce this "but substantial failure cases still remain"22 (§3.3.2).
| System | What is trained | Judge's training signal | Role in policy training | Reported |
|---|---|---|---|---|
| Self-Rewarding18 | the policy as judge (Llama 2 70B) | judging seed data, then DPO on own pairs | pairs for iterative DPO | judge agreement 65.1% to 81.7%; length 1,092 to 2,552 |
| J119 | separate judge, 8B / 32B / 70B | +1 correct verdict; +1 if correct in both orders (GRPO) | judge model | beats o1-mini, o3, R1-671B on some benchmarks (authors) |
| DeepSeek-GRM4 | separate 27B pointwise GRM | +1 / -1 for ranking the best response (GRPO) | reward model | 69.9, 71.0 (32 votes), 72.8 (meta-RM) on RM benchmarks |
| Kimi K2 critic9 | the critic, refined during RL | on-policy verifiable-reward rollouts | self-critic reward | qualitative only |
| Nemotron 3 Ultra GenRM22 | separate Ultra-based GenRM | RLVR-style rewards on scores and ranking | RLHF reward, principles vary by iteration | no standalone number |
| DeepSeek-V421 | the actor is the GRM | joint RL on rubric-guided data | reward for hard-to-verify tasks | claims only; no ablation |
7.4 What joint training changes
Four effects follow from the sources; the last three are my analysis. (1) Judge capacity has to track policy capacity: Nemotron reports that as policies grow they increasingly exploit weaker reward models, and responds by scaling up the GenRM22. (2) Judge and policy share failure modes when they share weights, so a quirk learned while generating can appear while judging, and nothing independent audits either. (3) The anchor moves from a dataset to a process: J1 and GRM are anchored by preference labels with known winners, K2 by verifier outcomes, V4 by "a minimal set of diverse human annotations" whose size the report does not give. (4) The monitor must sit outside the loop; the measurable quantities in the sources are judge accuracy on labelled pairs tracked through training (Self-Rewarding Table 4) and the Best-of-N versus pass@N gap (MiniMax-M1, Section 3.1). The only direct data point on a model judging its own family is DR Tulu's: Qwen3-8B as both rubric writer and judge captured +4.4 of the +5.7 average-point gain of GPT-4.1-class judges (Section 4.4), though that judge was fixed, not jointly trained15.
8. Process versus outcome judging at RL scale
Module 4 covers the reward-model evidence for process supervision: Lightman et al. found it outperforms outcome supervision for reward models on MATH (78% on a representative subset) and released PRM800K, 800,000 step labels23. The RL-scale reports then split by what the step-level judge is used for.
R1 lists three reasons it dropped process reward models: a fine-grain step is hard to define in general reasoning; judging whether an intermediate step is correct is hard, with automated annotation unsatisfactory and manual annotation unable to scale; and "once a model-based PRM is introduced, it inevitably leads to reward hacking", while retraining the PRM adds cost and complicates the pipeline. R1 grants that a PRM can rerank top-N responses or guide search, but finds its "advantages are limited compared to the additional computational overhead" in large-scale RL1 (G.2). Llama 3 shows the surviving use: outcome and stepwise reward models filter SFT reasoning traces "where the intermediate reasoning steps were incorrect", and Monte Carlo tree search with learned stepwise reward models generates valid traces for hard prompts24 (§4.3.3). GiGPO takes a third route for multi-turn agents: it builds step-level groups by finding repeated environment states ("anchor states") across trajectories, giving step-level relative advantages with no critic and no learned PRM, and reports gains over GRPO of more than 12% on ALFWorld and more than 9% on WebShop (Module 18)25.
The pattern is that step-level model judgments survive where nothing optimizes against them (data filtering, reranking, search) and are replaced inside the RL loop by outcome rewards with group-relative credit. The rubric and agentic judges of Sections 4-7 follow the outcome side: they score the final response against criteria, not each step of the reasoning.
9. What the judge costs, how reports route rewards, and how to choose
9.1 Judge calls per step
A judge-scored reward is a second inference workload inside the RL step. Its size follows from the scheme. R1's RL step used 32 questions with 16 samples each, 512 outputs1 (§3.2.1). Take \(P=32\) prompts and \(G=16\) rollouts, and RaR's average of \(k=7.5\) rubric items per prompt5:
- Pointwise, one call per rollout: \(PG=512\).
- Explicit rubric, one call per item: \(PGk=3{,}840\).
- Round-robin pairwise, one order: \(P\binom{G}{2}=3{,}840\); requiring agreement across both orders, as Zheng et al.'s conservative fix and J1's consistency reward do: \(PG(G-1)=7{,}680\).
- Single-elimination bracket: \(P(G-1)=480\), at the price of coarse ranks (at most \(\log_2 G+1\) distinct win counts).
Voting multiplies every line by the number of votes: DeepSeek-GRM scales to 32 samples4, and RLCF averages 25 samples per checklist item7. The calls are not equal in size: a judge prompt carries the task and the candidate or candidates, and Olmo 3's judge allows up to 32,768 input tokens against 2,048 response tokens6, a prefill-heavy shape that is ordinary serving work for you but a new line item in an RL budget. No report gives judge tokens per step. The evidence that it matters is indirect: RLCF reports its judge as the computational bottleneck7.
Judge calls per RL step
Formulas: \(nPG\), \(nPGk\), \(nP\,G(G-1)/2\), \(nP\,G(G-1)\), \(nP(G-1)\). Counts are judge calls, not tokens; multiply by the judge's reasoning length for cost.
9.2 How the reports route non-verifiable reward
| Report | Judge for non-verifiable tasks | Where criteria come from | Guard against exploitation |
|---|---|---|---|
| R1, Jan 20251 | preference reward models built from DeepSeek-V3 labels | Arena-Hard-style judge prompt | reward model active only in the last 400 of 1,700 steps; length-matched pairs |
| Qwen3, May 20252 | Qwen2.5-72B-Instruct with reference; preference RM without | per-query reference | rule-based rewards kept where possible |
| MiniMax-M1, Jun 20258 | own GenRM, vs ground truth or reference | reference answers | online length monitoring; GenRM recalibration |
| Kimi K2, Jul 20259 | self-critic, pairwise ranking | core, prescriptive and human rubrics | prescriptive rubrics; critic refined on verifiable rollouts; hack-check layer |
| DeepSeek-V3.2, Dec 202526 | generative reward model | per-prompt rubrics (author not stated) | not stated for the GRM; rules for reasoning and agent tasks |
| Olmo 3, Dec 20256 | Qwen3 32B LM judge, reference or open-ended | dataset references | F1 prompt filter; domain mixing |
| Kimi K2.5, Feb 202617 | GRMs on top of verified rewards | multiple alternative rubrics per task context | rubric rotation |
| GLM-5, Feb 20263 | rule, outcome-RM and generative-RM blend | not stated | blend of the three; human-written responses as stylistic anchors against "model-like" patterns |
| Nemotron 3 Ultra, Jun 202622 | principle-following GenRM, scaled up | principles adapted per RLHF iteration | larger GenRM; RLHF data mixed into verifiable stages |
| DeepSeek-V4, id 2606.1934821 | the actor as GRM; no scalar RMs | rubric-guided RL data | not stated |
| Kimi K3, Jul 202610 | agentic GRM, tournament-style group reward | the judge writes the rubric | verbosity guard; deterministic gate in web development |
The guards fall into six kinds: a hard rule inside the comparison (K3's length limit), a criterion inside the rubric (RLCF's universal items, K2's prescriptive rubrics), a veto (Rubicon's null score), an external monitor with judge recalibration (MiniMax-M1), data-side controls (R1's length-matched pairs), and a gate or anchor (K3's build check, K2's hack-check, GLM-5's human-written responses, added because purely model-generated optimization "tends to converge toward recognizably 'model-like' patterns"3, §3.4). Several reports name the exploit behind the guard (RLCF's preamble, Rubicon's praise and self-evaluation, MiniMax-M1's length, K3's verbosity); K2 and K2.5 cite reward hacking generically.
9.3 Choosing a source and naming its exploit
| Task property | Source | Exploit it invites | Countermeasure reported |
|---|---|---|---|
| A rule or test decides | verifier (Module 10) | loopholes in the rule | see Modules 10 and 19 |
| Answer exists, surface forms vary | reference-based judge | generic openers accepted as correct ("master keys"); ceiling at the reference's quality | negative-example data fix12; order swaps11; keep a rule check where possible2 |
| No reference; multi-criteria quality | rubric judge | rubric-specific padding and flattery: long preambles, opening praise, self-evaluation | universal criteria7; veto rubric16; evolving or rotated rubrics1517 |
| Open-ended preference at scale | pairwise or tournament judge | verbosity; order bias; a signal that never goes to zero, so judge noise is trained on (Section 6.3) | length guard10; swap agreement19; verifiable grounding of the critic9 |
| Last-mile polish after verifiable RL | scalar preference model, short exposure | reward rising while task performance falls | confine to a few hundred steps1; length-matched pairs1 |
| Judge capacity must grow with the policy | jointly trained GRM | shared weights, shared blind spots (my analysis) | external accuracy monitor; verifier-anchored slice (Section 7.4) |
Where experts disagree
The sources agree that verifiers beat judges where a verifier exists. They disagree about what to do where none does: keep models out of the reward, put a separate judge in, train the judge with the policy, or judge the reasoning steps.
A. Keep model judgment out of the RL reward where a program can decide
DeepSeek-R1 abstains from neural reward models for reasoning because they are hacked at scale, and confines its preference reward to the last 400 of 1,700 stage-2 steps1; Qwen3 keeps rule-based rewards for their "high precision"2.
Precision and an unmoving target matter more than coverage, and exposure to a model reward should be short.
- No judge drift and none of the master-key class of exploit.
- Cheap: no second inference workload (Section 9.1).
B. A separate rubric-guided judge is the general answer
Rubrics as Rewards5; Kimi K2, K2.5 and K391710; DeepSeek-V3.226; GLM-53; DR Tulu15.
Explicit criteria give interpretable, prompt-specific signal and let smaller judges do well; a rule-plus-judge blend covers tasks rules cannot.
C. Train the judge with the policy
DeepSeek-V4 (the actor is the GRM)21; Kimi K2's closed-loop critic9; precursors in Self-Rewarding LMs, J1 and DeepSeek-GRM18194.
Judge capacity and calibration should move with the policy, and judging is a verifiable task for the data that has labels.
- The judge improves instead of aging; DeepSeek claims minimal human annotation21.
- J1 and GRM show judges can be trained with the same GRPO machinery194.
- No ablation against a separate judge in the V4 or K2 reports.
- Self-judging models also grew their outputs (1,092 to 2,552, as printed)18; shared weights mean shared blind spots (my analysis).
D. Judge the reasoning steps, not just the answer
Lightman et al.23 for process reward models; opposed at RL scale by R11; used offline by Llama 324; replaced by anchor-state credit in GiGPO25.
Step-level labels localize errors that outcome rewards smear across a long trace.
- Process supervision beat outcome supervision for reward models on MATH23.
- Still useful for filtering SFT traces24.
- Hard to define a step, hard to verify one, and hackable once optimized against1.
Route by task property, as Table 7 does: a verifier where one exists, a reference-based judge where an answer exists but rules cannot read it, a rubric judge elsewhere, and a scalar preference model only for short exposure. A and B are complements: every report in Table 6 also uses rules or tests for its verifiable tasks. C is the direction of travel but rests on vendor statements; adopt it only with a judge-accuracy monitor on labelled pairs and a verifier-anchored slice outside the judge's control, because nothing in the sources audits a judge that shares weights with the policy. D belongs offline. The experiment that would settle B against C is an equal-compute comparison of a frozen and a jointly trained judge on the same policy and prompts, scored by humans or a verifier-anchored held-out set rather than by either judge, tracking judge accuracy on near-ties through training. None is in the sources.
What is changing
Three things are moving as of October 2026. Judges are becoming agents: K3 requires a read-rubric-score-record protocol and DeepSeek folds judging into the actor, so judging becomes a reasoning workload of its own1021. Rubrics are becoming dynamic: DR Tulu evolves them from rollouts and Rubicon grows a veto list from observed exploits1516. And the evidence base is still thin where it matters most: the K2, K2.5, V3.2, K3 and V4 judge designs are described, not ablated, and the V4 report (v1) and the K3 report (v2) may still change; V4's arXiv date metadata is inconsistent with its id. Reported numbers can also move with the evaluation judge itself; GLM-5 found BrowseComp scores sensitive to the judge prompt and model and standardized its judge3. Calibration, which K2's own limitation note ties to rubrics that penalize hedging, is Module 24.
References
- paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 v2 (4 Jan 2026). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: §2.2 (no neural reward models for reasoning), §3.1 (helpful and safety reward models, judge-labelled pairs), §3.2.2 and Eq. 8-10 (stage-2 reward mixture, preference reward in the last 400 of 1,700 steps), Appendix B.2 (judge prompt), B.3.3 and Listing 4 (generative reward model for SFT data), B.5 and Fig. 6 (reward hacking), G.1-G.2 (limitations, PRM failure modes).
- paperYang, A., Li, A., Yang, B., Zhang, B. et al. (Qwen Team, Alibaba). Qwen3 Technical Report. arXiv:2505.09388 v1 (14 May 2025). https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: §4.4 (General RL: more than 20 tasks, three reward types, why rule-based rewards are kept).
- paperGLM-5 Team (Zhipu AI, Tsinghua University). GLM-5: from Vibe Coding to Agentic Engineering. arXiv:2602.15763 v2 (24 Feb 2026); v1 17 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: §3.2 (judge models producing binary outcome rewards), §3.4 (hybrid rule/ORM/GRM reward system and human-authored anchors), §4.2.4 (judge-prompt and judge-model sensitivity of BrowseComp).
- paperLiu, Z., Wang, P., Xu, R., Ma, S. et al. (DeepSeek-AI, Tsinghua University). Inference-Time Scaling for Generalist Reward Modeling (DeepSeek-GRM, SPCT). arXiv:2504.02495 v3 (25 Sep 2025). https://arxiv.org/pdf/2504.02495. Accessed 2026-10-27. Used for: §2.1 and Eq. 1-3 (scalar, semi-scalar, generative; pointwise, pairwise), §2.2 and Table 1 (principles), §3 and Eq. 10-11 (rejective fine-tuning then rule-based GRPO, +-1 reward), §4 and Table 2 (voting and meta-RM results). Not in the researcher's source map; fetched for this module.
- paperGunjal, A., Wang, A., Lau, E., Nath, V. et al. (Scale AI). Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv:2507.17746 v2 (3 Oct 2025). https://arxiv.org/abs/2507.17746. Accessed 2026-10-27. Used for: §2 (Eq. 1-3, RLVR as the k = 1 case), §3 (rubric desiderata, 7-20 items, Essential/Important/Optional/Pitfall), §4 (setup: Qwen2.5-7B, GRPO, 16 rollouts, gpt-4o-mini judge), §5 and Tables 1-3, Fig. 2-3 (results and ablations), Table 11, Appendix A.1 (worked rubric).
- paperTeam Olmo (Allen Institute for AI). Olmo 3. arXiv:2512.13961 v2 (14 Apr 2026); v1 15 Dec 2025. https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: §4.4.1 (judge-based chat rewards in OlmoRL, Qwen3 32B judge settings, Fig. 16), §4.4.2 (chat prompt sourcing and filtering), §4.5 (mixed-domain RL and the LM-judge anchor).
- paperViswanathan, V., Sun, Y., Ma, S., Kong, X. et al. (CMU, Apple). Checklists Are Better Than Reward Models For Aligning Language Models (RLCF). NeurIPS 2025; arXiv:2507.18624 v2 (1 Dec 2025). https://arxiv.org/pdf/2507.18624. Accessed 2026-10-27. Used for: §2 (checklist desiderata, direct versus candidate-based extraction, universal criteria against preamble hacking, WildChecklists), §3 (25-sample judge average, program verifiers, 40% pair filter, DPO), abstract results. Not in the researcher's source map; fetched for this module.
- paperMiniMax. MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585 v1 (16 Jun 2025). https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-26. Used for: §4.2.1 (generative reward model as verifier; pairwise comparison against a reference answer scored -1/0/1; 25K general samples) and §4.2.2 (length bias of GenRMs, online monitoring and recalibration).
- paperKimi Team (Moonshot AI). Kimi K2: Open Agentic Intelligence. arXiv:2507.20534 v2 (3 Feb 2026); v1 28 Jul 2025. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-27. Used for: §3.2 (self-critic reward), §3.2.1 (hybrid rule verification with a hack-check layer; faithfulness judge; safety attack/target/judge), §3.2.2 (Self-Critique Rubric Reward, closed-loop critic refinement), Appendix F.1-F.3 (core and prescriptive rubrics, stated limitations).
- paperKimi Team (Moonshot AI). Kimi K3: Open Frontier Intelligence. arXiv:2607.24653 v2 (7 Aug 2026); v1 27 Jul 2026. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-27. Used for: §4.1.2 (Agentic Generative Reward Model: mandatory four-step protocol, tournament-style group reward with binary comparisons, verbosity guard; Reasoning Effort RL budget override) and §4.2.7 (web-development rewards: deterministic checks plus an internal reward model).
- paperZheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S. et al. (UC Berkeley and others). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks; arXiv:2306.05685 v4 (24 Dec 2023). https://arxiv.org/pdf/2306.05685. Accessed 2026-10-27. Used for: Abstract (over 80% agreement with humans), §3.3 and Tables 2-4 (position bias, repetitive-list verbosity attack, self-enhancement, math grading), §3.4 (swapping, few-shot, chain-of-thought and reference-guided judges). Not in the researcher's source map; fetched for this module.
- paperZhao, Y., Liu, H., Yu, D., Kung, S. Y. et al. (Princeton, Tencent AI Lab and others). One Token to Fool LLM-as-a-Judge. arXiv:2507.08794 v3 (11 Jun 2026). https://arxiv.org/pdf/2507.08794. Accessed 2026-10-27. Used for: Abstract and §1 (collapsed RLVR run with a Qwen2.5-72B-Instruct judge, Fig. 2), §3.3.1 and Table 1 (false positive rates of master keys), §3.3.2-§3.4 (Master-RM, CoT and no-question effects, Table 3). Not in the researcher's source map; fetched for this module.
- paperZheng, X., Pang, T., Du, C., Liu, Q. et al. (Sea AI Lab, SMU). Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates. ICLR 2025; arXiv:2410.07137 v2 (2 Mar 2025). https://arxiv.org/pdf/2410.07137. Accessed 2026-10-27. Used for: Abstract: a constant, instruction-independent output reaches an 86.5% length-controlled win rate on AlpacaEval 2.0, 83.0 on Arena-Hard-Auto and 9.55 on MT-Bench. Not in the researcher's source map; fetched for this module.
- paperSinghal, P., Goyal, T., Xu, J., Durrett, G. A Long Way to Go: Investigating Length Correlations in RLHF. COLM 2024; arXiv:2310.03716 v2 (10 Jul 2024). https://arxiv.org/pdf/2310.03716. Accessed 2026-10-27. Used for: Abstract and §3.1 (70-90% of PPO reward improvement explained by length on WebGPT and RLCD), §3.2 and Table 2 (a length-only reward comes close to PPO: 56% vs 58% and 64% vs 63%). Not in the researcher's source map; fetched for this module.
- paperShao, R., Asai, A., Shen, S. Z., Ivison, H. et al. (University of Washington, Allen Institute for AI and others). DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research. arXiv:2511.19399 v3 (15 May 2026). https://arxiv.org/pdf/2511.19399. Accessed 2026-10-27. Used for: §3.1 and Fig. 2 (search-augmented evolving rubrics, rubric buffer pruned by reward standard deviation), §4.3 and §5.1 (judge and rubric-generator models, 27,000 GPU hours), §6 and Tables 3-4, Fig. 6 (static versus evolving rubrics, random-reward control, open judge). Not in the researcher's source map; fetched for this module.
- paperHuang, Z., Zhuang, Y., Lu, G., Qin, Z. et al. (Inclusion AI, Ant Group, Zhejiang University). Reinforcement Learning with Rubric Anchors (Rubicon). arXiv:2508.12790 v1 (18 Aug 2025). https://arxiv.org/pdf/2508.12790. Accessed 2026-10-27. Used for: Abstract (10,000+ rubrics, +5.2% on open-ended benchmarks), §2 (veto, saturation-aware, pairwise-interaction aggregation), §3.1-§3.2 (stage-wise RL against the seesaw effect, adaptive reward-hacking defense), Appendix A.1 (defense rubric). Not in the researcher's source map; fetched for this module.
- paperKimi Team (Moonshot AI). Kimi K2.5: Visual Agentic Intelligence. arXiv:2602.02276 v2 (7 Aug 2026); v1 2 Feb 2026. https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-27. Used for: §4.4.2 (reward function; generative reward models deployed across chat, coding, search and artifact agents; multiple alternative GRM rubrics).
- paperYuan, W., Pang, R. Y., Cho, K., Li, X. et al. (Meta, NYU). Self-Rewarding Language Models. arXiv:2401.10020 v3 (28 Mar 2025). https://arxiv.org/pdf/2401.10020. Accessed 2026-10-27. Used for: §2 (LLM-as-a-Judge prompt, iterative DPO), §3 and Tables 1-4 (AlpacaEval 2.0 win rates, judge pairwise accuracy by iteration, response-length growth). Not in the researcher's source map; fetched for this module.
- paperWhitehouse, C., Wang, T., Yu, P., Li, X. et al. (FAIR at Meta). J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning. arXiv:2505.10320 v3 (13 Oct 2025). https://arxiv.org/pdf/2505.10320. Accessed 2026-10-27. Used for: Abstract and §2.1-§2.2 (judging recast as a verifiable task, 22K synthetic pairs, verdict-correctness and verdict-consistency rewards, GRPO). Not in the researcher's source map; fetched for this module.
- paperZhang, L., Hosseini, A., Bansal, H., Kazemi, M. et al. (Google DeepMind and others). Generative Verifiers: Reward Modeling as Next-Token Prediction. ICLR 2025; arXiv:2408.15240 v3 (22 Feb 2025). https://arxiv.org/pdf/2408.15240. Accessed 2026-10-27. Used for: Abstract: GenRM trained with the next-token objective and chain-of-thought verification; Best-of-N gain from 73% to 93.4% on GSM8K. Not in the researcher's source map; fetched for this module.
- paperDeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348 v1 (cited by title and id; the arXiv date metadata is inconsistent with the id). https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-27. Used for: §5.1.1 Generative Reward Model: no scalar reward models, rubric-guided RL data, RL applied to the GRM, the actor as the GRM, claimed benefit with minimal human annotation.
- paperNVIDIA. Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv:2606.15007 v1 (12 Jun 2026). https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Used for: §3.3.2 (chat teacher: principle-following GenRM trained with verifiable rewards; why larger policies exploit weaker reward models).
- paperLightman, H., Kosaraju, V., Burda, Y., Edwards, H. et al. (OpenAI). Let's Verify Step by Step. ICLR 2024; arXiv:2305.20050 v1 (31 May 2023). https://arxiv.org/abs/2305.20050. Accessed 2026-10-26. Used for: Abstract: process supervision outperforms outcome supervision for reward models on MATH; 78% of a representative subset; PRM800K.
- paperGrattafiori, A., Dubey, A., Jauhri, A. et al. (Llama Team, AI @ Meta). The Llama 3 Herd of Models. arXiv:2407.21783 v3 (23 Nov 2024). https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Used for: §4.3.3 (outcome and stepwise reward models to filter reasoning traces; MCTS with stepwise reward models).
- paperFeng, L., Xue, Z., Liu, T., An, B. Group-in-Group Policy Optimization for LLM Agent Training (GiGPO). NeurIPS 2025; arXiv:2505.10978 v3 (28 Oct 2025). https://arxiv.org/abs/2505.10978. Accessed 2026-10-26. Used for: Abstract: anchor-state grouping gives step-level relative advantages without a critic.
- paperDeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556 v1 (2 Dec 2025). https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-27. Used for: §3 Mixed RL Training (rule-based outcome reward, length penalty and language consistency for reasoning and agent tasks; generative reward model with per-prompt rubrics for general tasks).