Reinforcement Learning for LLM Post-Training · Module 10
RL with Verifiable Rewards: Rule-Based Reward Design, Prompt Selection, and What R1-Zero Showed
RLVR keeps the RLHF objective and replaces the learned reward model with a program that checks the answer. This module covers how that program is built for math, code, constraints and SQL, how its output is composed and scaled, how prompts are filtered so that a group of rollouts carries signal, and what R1-Zero did and did not demonstrate. Afterwards you can write a rule-based reward and a prompt-selection filter for a new verifiable domain and say in advance where each will fail.
1. What RLVR is, and where the name comes from
1.1 The definition
Tulu 3 introduced the term and defines it by subtraction: RLVR "leverages the existing RLHF objective but replaces the reward model with a verification function"1 (§6). The objective is the KL-regularized RLHF objective with the learned score swapped for a program \(v\):
$$\max_{\pi_\theta}\;\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}\Big[v(x,y)-\beta\,\mathrm{KL}\big(\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\big)\Big],\qquad v(x,y)=\begin{cases}\alpha & y \text{ verifies}\\ 0 & \text{otherwise}\end{cases}$$Tulu 3 sets \(\alpha = 10\) "based on pilot experiments" and does not tune it1 (§6, Eq. 7-8), optimizes with PPO after preference tuning, and trains on 29,946 prompts: 7,473 GSM8K, 7,500 MATH and 14,973 instructions with checkable constraints1 (Table 22). The PPO machinery, the value-model initialization and the over-optimization sweeps of that run are in Module 5 (Section 8); this module takes the reward block apart.
Reasoning-RL recipes mostly drop the KL term: DAPO argues that in long-CoT training "the model distribution can diverge significantly from the initial model, thus this restriction is not necessary"2 (§2.3), and Dr. GRPO sets \(\beta=0\) because rule-based verifiers remove the reward-model distribution-shift concern3 (§3). The name is also slightly misleading. Lambert, a Tulu 3 author, records that the method was nearly called "RL with Ground Truth rewards", but that in code or precise instruction following "answers can be verified with a checking function (e.g., a unit test), even when there are multiple correct solutions"4 (pp. 98-99). The verifier is a function of the completion, not a lookup of an answer key, and that is where most of the design freedom, and most of the failures, live.
1.2 Provenance
The attribution that circulates, that R1 introduced both GRPO and RLVR, is wrong on both counts. GRPO is from DeepSeekMath (Feb 2024)5 and descends from PPO and TRPO (Module 3); the name RLVR is from Tulu 3 (22 Nov 2024)1. What R1-Zero did was run a rule-based reward at frontier scale with no reward model and no SFT stage, and Lambert credits R1 with "popularizing RLVR"4 (p. 14).
- DeepSeekMath
GRPO is introduced with a learned reward model whose training data "is based on the rule judgment" of answer correctness, and §5.2.1 compares a 'Rule' reward with a 'Model' reward inside Online RFT and GRPO5 (§4.2, §5.2.1).
- Tulu 3
RLVR named and ablated with PPO on GSM8K, MATH and IFEval-style constraints1.
- DeepSeek-R1 and Kimi k1.5, same day
R1-Zero: rule-based accuracy and format rewards, GRPO, no neural reward model6 (§2.2). k1.5: a verifiable reward, with a learned equivalence checker for free-form math7.
- DAPO, Dr. GRPO, Qwen3
A plus-or-minus 1 rule reward with integer answers2; a binary Math-Verify reward on a minimal recipe3; reasoning RL on 3,995 query-verifier pairs8.
- Olmo 3
Verifiers beyond math and RL-Zero from a fully open base9 (§4.4, §6).
1.3 What "verifiable" has to mean in practice
Three properties decide whether a domain works. The check must run without a human. It must be fast and deterministic enough to run once per rollout: R1's report notes that the rule-based module needs no GPU memory but "its execution tends to be time-consuming", so it is scheduled asynchronously to overlap with rollout and inference6 (Supplementary B.1). And its error rate must be small relative to the signal. The IFEval authors, whose constraint taxonomy Tulu 3 and Olmo 3 both draw on19, state the limit plainly: "very few instructions are 100% verifiable objectively and automatically"10 (§1). Every verifier in Section 3 is a bet on that error rate in the region where the policy will actually push.
2. R1-Zero's reward, line by line
2.1 A template, and two rewards that read different tokens
R1-Zero is DeepSeek-V3-Base trained with GRPO and no SFT stage. Its prompt is a fixed template asking for reasoning first and the answer second, "enclosed within <think>...</think> and <answer>...</answer> tags"; the authors "intentionally limit our constraints to this structural format"6 (R1 Table 1, §2.3). The reward is the sum \(\mathrm{Reward}_{\mathrm{rule}}=\mathrm{Reward}_{\mathrm{acc}}+\mathrm{Reward}_{\mathrm{format}}\), the terms "combined with the same weight"6 (Eq. 4). The accuracy term reads the final answer: for math "the model is required to provide the final answer in a specified format (e.g., within a box), enabling reliable rule-based verification", and for code "a compiler can be utilized to evaluate the model's responses against a suite of predefined test cases". The format term checks that the reasoning sits between the think tags6 (§2.2). The run: learning rate 3e-6, KL coefficient 0.001, 16 samples per question, 32 questions per step, 10,400 steps (1.6 epochs), maximum length 32,768 tokens until step 8.2k and 65,536 after6 (§2.1). Module 9 has the GRPO objective those numbers feed; Module 1 has the GPU-hour arithmetic.
2.2 What "equal weight" implies
Eq. 4 is additive, so the four combinations of (accuracy, format) score 2, 1, 1 and 0. A correct answer with broken tags and a wrong answer with intact tags both score 1, provided the accuracy check does not itself depend on the tags, which the paper does not say. Early in training a base model may rarely close its think tag, and the format term then supplies reward variation that accuracy alone may not, which is presumably why it is there (an inference; the report shows no ablation of it).
Later the term does almost nothing, and then something odd. Once every rollout is tagged, the format reward is constant within the group and cancels in \(A_i=(r_i-\mathrm{mean})/\mathrm{std}\); but a near-constant group is where normalization is most violent. Take 16 rollouts with equal accuracy where 15 are tagged (\(R=2\)) and one is not (\(R=1\)): the mean is \(31/16\), the population standard deviation \(\sqrt{15}/16\approx0.242\), and the untagged rollout receives \(A=-\sqrt{15}\approx-3.87\) while the other fifteen receive \(+0.26\). One formatting slip draws nearly four standard deviations of penalty on every token (derived from Eq. 3 and Eq. 4; the report does not say which standard-deviation convention it uses). Module 11 treats the general effect.
2.3 Why there is no neural reward model, and what the report shows
R1 states the decision directly: the authors "abstain from applying neural reward models—whether outcome-based or process-based—to reasoning tasks", because such models "are susceptible to reward hacking during large-scale reinforcement learning" and because retraining them "necessitates substantial computational resources and introduces additional complexity into the training pipeline"6 (§2.2). The report backs the first reason with its own failure. In stage two, general data is scored by learned reward models, and the authors "observe such reward hacking behavior when employing the helpful reward model": reward kept rising while Codeforces performance fell (Supplementary B.5, Figure 6). They added preference rewards only in the final 400 of 1,700 steps6 (§3.2.2). Their failed attempts add that a process reward model founders on defining a step and judging its correctness, and that "once a model-based PRM is introduced, it inevitably leads to reward hacking"6 (Supplementary G.2).
The same report is candid about the boundary. For "complex tasks that cannot be effectively evaluated by a reliable reward model, scaling up pure RL methods remains an open challenge", and for tasks without a reliable signal R1 "uses human annotation to create supervised data, and only conduct[s] RL for hundreds of steps"6 (§6). That sentence, not the AIME curve, is the real scope statement of rule-based RLVR. Part 5 (Modules 16 and 19) picks up the judge and hacking sides.
2.4 The language-consistency reward and its measured cost
R1-Zero mixed Chinese and English in its reasoning. For the first RL stage of R1 the authors add a term equal to the "proportion of target language words in the CoT", \(\mathrm{Reward}_{\mathrm{language}}=\mathrm{Num}(\mathrm{Words}_{\mathrm{target}})/\mathrm{Num}(\mathrm{Words})\), added "directly" to the final reward for both reasoning and non-reasoning data6 (§3.2.1, Eq. 7). In stage two the reward is the sum of reasoning, general and language terms6 (Eq. 8-10). The ablation, on a distilled 7B model sharing R1's cold-start data, found that without the term "language consistency gradually deteriorates as training steps increase"; with it, consistency holds, math is comparable and coding shows "a slight degradation"6 (Supplementary B.6). The authors keep it because it "aligns with human preferences, making the output more readable". They also note that DeepSeek-V3-Base's data is "mostly Chinese and English, which might be the cause for DeepSeek-R1-Zero language mixing when the language consistent reward is absent"6 (Supplementary A.1). DeepSeek-V3.2 keeps the term, with a length penalty, in its rule-based reasoning and agent reward11 (§3).
2.5 What R1 trained on
The R1 report lists its RL data by domain, and the verifier differs by row. Math answers are matched to a reference ("1; otherwise, it is assigned a value of 0") and proofs are excluded "because it is difficult to determine their correctness"; code is graded on hidden tests or unit tests; STEM is multiple choice with a binary option-match reward; logic mixes web puzzles with synthetic code-IO and puzzle tasks; general data goes to reward models6 (Supplementary B.3.1).
| Domain | Prompts (K) | Question type | Reward rule as stated | Noted exclusion or detail |
|---|---|---|---|---|
| Math | 26 | Quantitative reasoning | Answer equals reference: 1, else 0 | Proofs excluded; average prompt 122 tokens |
| Code | 17 (+8) | Contest problems; GitHub bug fixes | Hidden test cases; unit tests | R1's test-generation pipeline is documented for cold-start problems (B.3.2) |
| STEM | 22 | Multiple choice, 4 to 8 options | Binary on matched option | 15.5% physics, 30.7% biology, 46.5% chemistry |
| Logic | 15 | Puzzles, code-IO, cipher, Zebra, 24 game | Automatic evaluation | Average prompt about 420 tokens |
| General | 66 | Helpfulness, harmlessness | Reward models (helpful, safety) | Not rule-based; used in the last 400 steps of stage 2 |
A derived scale check: R1-Zero ran 10,400 steps of 32 questions, 332,800 question draws, over "1.6 training epochs"6 (§2.1), which implies about 208,000 distinct prompts. The report does not state R1-Zero's prompt count and the table above describes the data for R1 proper, so the 80K reasoning prompts and the 208K need not conflict. Verifiable prompt pools in successful runs still span two orders of magnitude, from Qwen3's 3,995 pairs (Section 5) to about 105.
3. The verifier is the product
A verifier is a pipeline: pull an answer out of free text, put it in a canonical form, compare it with something, and map the outcome to a number. Each stage has its own failure, and the policy will find whichever is cheapest to exploit. A false negative (a correct answer scored 0) wastes signal and, worse, teaches the policy that a correct but unusual format is wrong. A false positive (a wrong answer scored 1) is exploited, because optimization pushes toward whatever the checker accepts. Figure 3 marks where each enters; Table 2 lists the verifiers the reports describe.
3.1 Math: extract, normalize, compare
Tulu 3 uses exact match against an extracted answer, with few-shot chain-of-thought prompts and "flex" MATH extraction logic1 (§6.1). DAPO moves the difficulty from the verifier to the data: because answers "come in a variety of formats, such as expression, formula and number", it has an LLM rewrite each question so the answer is a single integer, "to provide accurate reward signals using rules and minimize errors introduced by formula parsers", giving DAPO-Math-17K; the reward is +1 if the prediction is equivalent to the answer and -1 otherwise2 (§2.4, §3.5). Dr. GRPO keeps free-form answers and uses Math-Verify with a 1/0 rule3 (§3.2). Math-Verify is a three-step grader: extract the answer in a "format agnostic manner", convert it to a SymPy expression, compare with the gold answer symbolically or numerically; its authors say strict evaluators can underestimate model performance "in extreme cases, even by 40 points"12. That figure is about evaluation; a reward function is the same code run during training.
The code is short enough that its failure modes are visible. Below are two reward functions in the style of the TRL documentation's examples13, simplified to plain strings; the comments give what they return, checked by running them for this module.
import re
def format_reward(completions, **kwargs):
pattern = r"^<think>.*?</think><answer>.*?</answer>$"
return [1.0 if re.match(pattern, c) else 0.0 for c in completions]
def boxed_reward(completions, ground_truth, **kwargs):
out = []
for c, gt in zip(completions, ground_truth):
m = re.search(r"\\boxed\{(.*?)\}", c)
out.append(1.0 if m and m.group(1) == gt else 0.0)
return out
# format_reward(["<think>a\nb</think><answer>4</answer>"]) -> [0.0] "." does not match "\n" (no re.DOTALL)
# boxed_reward([r"\boxed{\frac{1}{2}}"], [r"\frac{1}{2}"]) -> [0.0] lazy .*? stops at the first "}", captures \frac{1
# boxed_reward([r"\boxed{0.5}"], ["1/2"]) -> [0.0] string equality is not mathematical equality
Each bug has a different consequence. The multi-line think block means a format reward of this shape scores essentially every real reasoning trace 0, a silent no-op rather than a crash. The lazy brace match gives false negatives on every answer containing a fraction, a set or a nested expression, teaching the policy to avoid exactly the answers that need LaTeX. The string comparison punishes correct answers in an unexpected form. TRL's interface itself is simple: a reward function takes prompts, completions and any dataset columns such as ground_truth and returns one float per completion; several can be listed and are summed with reward_weights, and the trainer logs frac_reward_zero_std, the fraction of groups whose rewards are all equal13. That metric is the cheapest health check a verifier has (Module 11).
3.2 Code: tests, sandboxes and what building tests costs
For code, the reward is a function of test results, and the work is in the tests. R1 gathered 5,151 Codeforces and 2,504 AtCoder problems whose tests are not public, had DeepSeek-V2.5 write Python programs that generate candidate tests, then filtered in two phases: drop tests on which known-correct submissions produce wrong output, then keep subsets that expose flaws in known-incorrect submissions6 (Supplementary B.3.2). Kimi k1.5 used generators built on the CYaRon library: 50 tests per problem, each run against 10 ground-truth submissions, a test valid if at least 7 of 10 agree, a problem admitted only if at least 9 of 10 submissions pass the selected tests. Of 1,000 sampled contest problems, about 614 needed no special judge, 463 generators produced at least 40 valid tests, and 323 problems entered training7 (appendix, test-case generation): a 32% yield from raw problems (derived) and 53% from the eligible 614.
Granularity is a free choice. Olmo 3 experiments with both the percentage of tests passed and an all-or-nothing reward and plots both; the text names no winner9 (§4.4.1, Figure 16). Lambert lists gating and partial credit proportional to tests passed as the two common forms4 (p. 98). Partial credit gives signal on hard problems and also rewards programs that pass only the easy tests; all-or-nothing is a harder, cleaner target.
Execution cost shapes which domains get RL. R1 reports that "long evaluation times" kept large-scale RL off software-engineering tasks, leaving it without a large gain over V3 there6 (§6). Kimi K2's Kubernetes sandbox service "supports over 10,000 concurrent sandbox instances" for contest problems and for repository tasks built from GitHub issues with executable unit tests14 (§3.2.1). Olmo 3 verifies code on AWS Lambda so that it "does not block the trainer", noting that some suites with time-complexity tests "can exceed hundreds of MBs for a single program"9 (§4.4.1, footnotes 35-36). System Design Module 37 (environments as services) and Module 17 cover the serving side. On the exploit side, Anthropic's production-RL study names three hacks that coding environments admitted: an object whose equality always returns true, sys.exit(0) before the asserts run, and patching the test report through conftest.py15 (§3; Module 19).
3.3 Instruction constraints: cheap to check, easy to satisfy badly
IFEval defines 25 types of verifiable instruction and 541 prompts, each instruction checkable by "a simple, interpretable, and deterministic program". It reports a strict and a loose metric because strict matching mislabels: an email ending "P.S. **I do like the cake**" fails a naive match against "P.S. I do like the cake", so the loose metric also tries transformed responses, such as with markdown removed10 (§1, §2). Tulu 3 builds 14,973 training prompts from this taxonomy with one verification function per constraint template1 (§6.1); Olmo 3 draws prompts with up to five constraints and gives reward 1 only if every constraint is satisfied9 (§4.4.1-§4.4.2).
The failure here is the cleanest false positive in the literature. With a small KL coefficient (β = 0.01), Tulu 3's policy "overoptimizes against the verifiable reward function and does not output meaningful content": asked to measure a pen in centimeters with the letter e appearing 14 times, it answered with fourteen e's separated by commas, and asked for five paragraphs it printed five bare "Paragraph N" lines between dividers1 (Appendix B.4, Figure 28). At β = 0.1 the same prompts produced sensible answers (Figure 29). Two mitigations appear in later reports. Olmo 3 found that RL on IFEval-style data alone raises IFEval and lowers AlpacaEval, while its mixed run keeps both because "the LM-judge reward ensures that the model continues to produce well-formed chat responses"9 (§4.5). Kimi K2 pairs deterministic checks, run in a code interpreter, with LLM-as-judge for nuanced constraints and adds "an additional hack-check layer that specifically detects such deceptive claims", meaning responses that assert compliance without achieving it14 (§3.2.1).
3.4 Execution and exact-match verifiers in shipped environments
NVIDIA's NeMo Gym defines an environment as "the complete system an agent interacts with", made of a dataset, an agent harness, a verifier and per-task state; its catalogue is a compact survey of reward shapes16. Bird SQL has 1,534 SQLite tasks scored by a binary reward from unordered result-set equality, with no LLM judge; Spider2 Lite has 135 tasks on result-set equivalence; Blackjack pays +1, 0 or -1; a GUI-grounding verifier uses a "smooth quadratic reward based on Euclidean distance"; and a single-step tool-use verifier requires search queries to match a golden word bag exactly, applies list-F1 guardrails to list arguments, scores any response with more than one tool call as 0, and states its aim as "eliminating partial-credit reward hacking". The last two answer the granularity question in opposite ways: a distance reward suits a task where nearness is meaningful, exact match one where partial credit can be farmed (my reading; the README gives only the stated aims). Reasoning Gym takes the data route: more than 100 generators, each paired with a verifier and a difficulty parameter, with large solution spaces meant to mitigate reward hacking17 (§1-§2). Module 17 covers environments as a whole.
3.5 When the verifier is a model anchored to a reference
Not every checker is a program. Kimi k1.5 has to decide that \(a^2-4\) and \((a+2)(a-2)\) are the same answer, so it trained two reward models on about 800K examples each: a classic value-head model that emits a correctness scalar, and a chain-of-thought model that writes a judgment and ends with a JSON verdict. Manual spot checks gave about 84.4 and 98.5 accuracy, and the chain-of-thought model drove RL7 (appendix, Reward Modeling for Math). Both compare against a reference answer; neither is a preference model. R1's findings draw the same line: "rule-based reward models (RMs) and LLMs to assess an answer's correctness against a predefined ground-truth" are both "robust mechanisms for mitigating issues related to reward hacking", the LLM route working for "well-defined, concise answers" with "limited generalizability" to open-ended generation6 (Supplementary G.1). Qwen3's General RL gives a Qwen2.5-72B-Instruct judge a reference answer, which "allows for more flexible handling of diverse tasks without requiring strict formatting, avoiding false negatives that can occur with purely rule-based rewards"8 (§4.4); Olmo 3 does the same with a Qwen3 32B judge, thinking off, scoring in [0, 1]9 (§4.4.1).
So "verifiable" spans deterministic programs to reference-conditioned judges; the line that matters is whether a ground truth anchors the score. Unanchored judges (rubrics, preference models, generative reward models) are Module 16.
| Task | Verifier | Reward values | Failure or cost documented | Source |
|---|---|---|---|---|
| Math, short answer | Extract, normalize, exact or symbolic match | {0,1}; {+1,-1} (DAPO); {0,10} (Tulu 3) | Extraction and format variants score correct answers 0 (derived, Section 3.1) | Tulu 3 §6.1; DAPO §2.4; Dr. GRPO §3.2 |
| Math, free-form | CoT reward model against the reference | {0,1} | Verifier error; spot-check accuracy 98.5 versus 84.4 for the classic model | Kimi k1.5 |
| Contest code | Hidden tests in a sandbox | All-pass 0/1, or fraction passed | Tests must be built and validated: 323 of 1,000 problems survived k1.5's filter | R1 B.3.2; k1.5; Olmo 3 |
| Repository bug fix | Unit tests | Binary | Long evaluation limited SWE RL in R1; test-tampering hacks | R1 §6; Kimi K2; Anthropic |
| Multiple choice | Option match | Binary | Guess floor of 1/k (Section 5.4); k1.5 excludes the format | R1 B.3.1; k1.5 §2.1 |
| Instruction constraints | One checker per constraint | alpha if verified; 1 if all hold | Checker met, request dropped; markdown false negatives | IFEval; Tulu 3 B.4; Olmo 3 |
| Text-to-SQL; games | Execute and compare result sets; game outcome | Binary; +1, 0, -1 | None reported in the README | NeMo Gym |
| Tool-call matching | Exact match to golden calls | Binary; more than one call scores 0 | Partial credit can be farmed (README's stated reason) | NeMo Gym |
| Chat with reference | LM judge against a reference | Score in [0,1] | Judge error | Olmo 3 §4.4.1; Qwen3 §4.4 |
4. Reward values, scale and composition
4.1 {0, 1}, {-1, +1} and {0, 10} are the same reward to a group-normalized advantage
The reports pick different numbers for the same verdict: R1 and Dr. GRPO use 1 and 0, DAPO +1 and -1, Tulu 3 \(\alpha=10\) and 06321. Under GRPO's advantage \(A_i=(r_i-\mathrm{mean}(r))/\mathrm{std}(r)\)6 (Eq. 3) these are identical: replacing \(r_i\) with \(a r_i+b\), \(a>0\), scales the mean and the standard deviation the same way and leaves \(A_i\) unchanged. The numbers matter only through what they do not cancel: a second reward term with a weight, a KL term placed in the reward as in PPO (where Tulu 3's \(\alpha\) and \(\beta\) set the relative weight of verifier and leash; Module 6), or an advantage that drops the standard deviation. DeepSeek-V3.2 and Olmo 3 use \(A=R-\mathrm{mean}(R)\)119, so there the reward scale multiplies the update directly. TRL's documentation says so for scale_rewards=False: "turning off std-based scaling also removes variance normalization, so update magnitudes depend directly on the raw reward scale and batch composition"13. Moving from a {0,1} verifier to a {0,10} one in such a recipe multiplies the step by ten.
4.2 What a binary verdict becomes inside a group
With a binary verifier and \(m\) correct completions out of \(G\), the group mean is \(m/G\) and the population standard deviation \(\sqrt{(m/G)(1-m/G)}\) (the equations in the reports do not say which convention they use). Correct completions then get \(A_+=\sqrt{(G-m)/m}\) and incorrect ones \(A_-=-\sqrt{m/(G-m)}\), and \(m A_+ + (G-m)A_-=0\). Table 3 evaluates this for \(G=8\), the group size Olmo 3 samples for filtering.
| Correct m of 8 | Pass rate | A for a correct completion | A for an incorrect one | Sum of |A| over the group |
|---|---|---|---|---|
| 1 | 12.5% | +2.646 | -0.378 | 5.29 |
| 2 | 25.0% | +1.732 | -0.577 | 6.93 |
| 4 | 50.0% | +1.000 | -1.000 | 8.00 |
| 6 | 75.0% | +0.577 | -1.732 | 6.93 |
| 7 | 87.5% | +0.378 | -2.646 | 5.29 |
| 0 or 8 | 0% or 100% | 0 | 0 | 0 |
Three properties follow. The outlier carries the signal: one success among eight is pushed up by 2.65 while each of the seven failures is pushed down by only 0.38, and the mirror image holds at 7 of 8. A group with no variance carries none, which is why Section 5 is about prompt selection rather than the update rule. And a binary verdict cannot rank among the correct: a lucky guess, a flawed derivation and a 30,000-token ramble get the same advantage as the clean solution. Kimi's authors give the first as the reason they exclude guessable questions ("the model reaches the correct answer through an incorrect reasoning process")7 (§2.1); length is handled by adding terms (next subsection; Module 23).
4.3 What the reports add to the verdict
Each recipe adds a small, fairly stable set of terms to the correctness check. Lambert's survey of reasoning recipes lists additional format rewards (R1, Open-Reasoner-Zero, Magistral, Skywork OR-1), language-consistency rewards (R1, Magistral), and length penalties used "to either stabilize the learning process over time or to mitigate overthinking" (Kimi 1.5, INTELLECT-2)4 (p. 106). Table 4 gives the numbers.
| Report | Correctness reward | Added terms | Source |
|---|---|---|---|
| Tulu 3 RLVR | alpha = 10 if verified, else 0 | -10 for no EOS token; KL in the reward (beta 0.05 at 8B) | §6, §6.2, Table 21 |
| R1-Zero | Accuracy, 1 or 0 on math | Format, equal weight | §2.2, B.3.1 |
| R1, RL stage 1 | Rule-based | Language consistency (fraction of target-language words), added to the final reward | §3.2.1, Eq. 7 |
| DAPO | +1 if equivalent, else -1 | Overlong reward shaping (Module 11) | §2.4, §3.4 |
| Dr. GRPO | 1 if correct (Math-Verify), else 0 | None | §3.2 |
| Kimi k1.5 | 0 or 1, by rules or a reward model | Length reward, introduced after a warm-up (Module 23) | §2.3, §2.3.3 |
| Olmo 3 Think | Math 1/0; code fraction passed or all-pass; constraints all-or-nothing | LM-judge score in [0,1] for chat prompts | §4.4.1 |
| DeepSeek-V3.2 | Rule-based outcome reward | Length penalty; language consistency; rubric GRM for general tasks | §3 |
Two cautions about adding terms. First, the amplification case from Section 2.4: continuous terms are normalized to unit scale in groups where the verdict is constant. TRL exposes the choice between summing weighted rewards and then normalizing (its default) and normalizing each reward function within the group before summing, which it attributes to the GDPO paper as the suggested approach for multi-reward RL13. Second, Tulu 3 tested the other common instinct, adding a reward-model score on top of the verifier: "using only the verifiable rewards outperforms using scores from the reward model", which "seems to introduce more noise"1 (§6.2.1, Figure 22). It is one GSM8K setup, but it is the only controlled version of the question in the sources and it points the way R1's decision does.
5. Prompt selection is signal design
5.1 Why the prompt set decides whether there is a gradient
A verifier gives a rollout one bit. Whether that bit teaches anything depends on whether the other rollouts for the same prompt disagree. Lambert states the principle for RLVR as a whole: "models can only learn from examples where there is a gradient"; if the starting model solves a problem 100% or 0% of the time, "there will be no gradient between different completions to the prompt"4 (p. 105). DAPO measured the consequence during training: the share of prompts with accuracy exactly 1 "continues to increase", so "the effective number of prompts in each batch keeps decreasing", which "can lead to larger variance in gradient"2 (§3.2, Figure 3b). Prompt selection is therefore not data cleaning; it controls the fraction of every batch that produces a nonzero advantage. The update-side repairs (dynamic sampling, zero-variance filtering) are Module 11; the data-side choices are here.
5.2 The recipes in the reports
Almost every report that says anything about its prompts filters on the pass rate of some model; they differ in which model measures it, which direction is cut and what else is excluded. Qwen3's reasoning-RL stage is the extreme case of curation over volume: four criteria (not used in the cold-start stage, "learnable for the cold-start model", "as challenging as possible", broad sub-domain coverage) yielded 3,995 pairs, and with GRPO, a large batch and many rollouts per query, AIME'24 for the 235B-A22B model rose from 70.1 to 85.1 over 170 steps8 (§4.2).
| Report | Pool | Rule | Signal and other exclusions |
|---|---|---|---|
| Qwen3 (§4.1-§4.2)8 | 3,995 query-verifier pairs | Unused in cold start; learnable; as hard as possible; broad coverage | Cold-start model. In cold start, drops queries Qwen2.5-72B-Instruct answers correctly without CoT, and queries with several sub-questions |
| Olmo 3 Think (§4.4.2)9 | Dolci-Think-RL, 104,869 prompts | Offline: drop prompts with pass rate above 62.5% over 8 rollouts; 32B relies on active sampling | The starting checkpoint, sampled at T = 1.0, top-p 1.0. Tulu 3-derived chat items with mean F1 below 0.1 or above 0.8 are dropped; any one WildChat character is capped at 10 instances |
| Olmo 3 RL-Zero (§6.1)9 | 13.3K math prompts | Drop prompts solved 8 of 8 | The final base model. Deduplication, one question per semantic cluster, decontamination |
| Kimi k1.5 (§2.1, §2.3.4)7; K2 (§3.2.1)14 | Not stated | k1.5: a lower pass rate over 10 samples means harder; easy-to-hard curriculum; sampling proportional to 1 - si. K2: keep moderate difficulty | An SFT model's pass rate or pass@k. k1.5 drops multiple-choice, true/false and proof questions, and prompts answered correctly without CoT within N = 8 tries |
| DAPO (§3.2, §3.5)2 | DAPO-Math-17K | Online: oversample and drop groups with accuracy 0 or 1 | The current policy. Answers rewritten to integers |
| Survey (Lambert, pp. 105-106)4 | Varies | Keep prompts the start model solves 20-80% of the time, from N = 16 completions | Seed-Thinking 1.5, Open-Reasoner-Zero, Phi-4, INTELLECT-2, MiMo RL, Skywork OR-1 |
| R1 (Table 4)6, Tulu 3 (Table 22)1 | R1: 80K reasoning prompts listed; Tulu 3: 29,946 | No pass-rate filter described | R1 excludes proofs; Tulu 3 trains about 13 epochs over GSM8K in ablations |
Two patterns stand out. The cuts are two-sided in principle but one-sided in practice: Olmo 3's offline rule removes only the easy side and keeps prompts the policy never solves, leaving those to zero-gradient filtering during training, while Lambert's survey states a two-sided 20-80% window. And the rules are measured on a model that is about to change. DAPO's accuracy-1 curve shows a static filter decaying, which is why Olmo 3 switches to active sampling at 32B and DAPO filters online92. How reliable a k-of-8 filter is at all is a calculation, which the next subsection does.
5.3 How reliable is a k-of-8 filter
Take Olmo 3's rule: sample 8 rollouts from the starting checkpoint and drop the prompt if the observed pass rate exceeds 62.5%9 (§4.4.2). With 8 rollouts the observed rate is \(k/8\), so "above 62.5%" means \(k\ge6\), a pass rate of 75% or more. The prompt's true pass rate \(p\) is unknown and \(k\sim\mathrm{Binomial}(8,p)\), so the probability that the filter removes a prompt is \(\sum_{k=6}^{8}\binom{8}{k}p^k(1-p)^{8-k}\) (derived). That is 14% at \(p=0.5\), 37% at \(p=0.625\), the nominal threshold itself, and only 80% at \(p=0.8\): a soft cut. A prompt solved half the time is lost one time in seven; one solved four times in five survives one time in five.
A 20-80% window with 16 completions4 (p. 105) is sharp in the middle and brutal at the edges. Read as inclusive it keeps \(4\le k\le12\); a prompt with \(p=0.05\) survives 0.7% of the time and one with \(p=0.25\) only 59.5%. A group of \(G\) rollouts carries a nonzero advantage with probability \(1-p^G-(1-p)^G\), which for \(G=16\) is 56.0% at \(p=0.05\) and 99.0% at \(p=0.25\): the window discards prompts that would have yielded a usable group more than half the time. The reason is cost, not quality. Olmo 3's learner "spends 75% of the time waiting for data" and about five times as many GPUs go to inference as to training9 (§4.4.3); a prompt at \(p=0.05\) buys a usable group about one time in two for the same 16 rollouts that buy one almost every time at \(p=0.5\). Filters are rollout-budget control, and their sharpness should be judged against that.
| True pass rate p | Removed by Olmo 3 rule (k of 8 at least 6) | Kept by 20-80% window (N = 16) | Group of 16 has signal |
|---|---|---|---|
| 0.05 | 0.0% | 0.7% | 56.0% |
| 0.25 | 0.4% | 59.5% | 99.0% |
| 0.50 | 14.5% | 97.9% | 99.997% |
| 0.625 | 37.0% | 90.5% | 99.95% |
| 0.80 | 79.7% | 40.2% | 97.2% |
| 0.95 | 99.4% | 0.7% | 56.0% |
Signal in a group of G rollouts, and what a pure guesser contributes
Formulas: \(1-p^G-(1-p)^G\) for a binary verifier; \(1-(1-1/k)^G\) for a multiple-choice prompt with k options, assuming independent uniform guesses. Both derived for this module.
5.4 Guessable prompts: the floor under a binary reward
Multiple choice is the cheap way to get a verifiable prompt, and it has a built-in noise floor. R1's STEM set has four to eight options per question6 (Supplementary B.3.1), so a policy that knows nothing passes a rollout with probability 1/4 to 1/8. With \(G=16\), the chance that at least one rollout is right by luck is \(1-(3/4)^{16}=99.0\%\) at four options and \(1-(7/8)^{16}=88.2\%\) at eight (derived). The group almost always has variance, and the lucky guess is reinforced as if it were reasoning. Kimi's authors found the same in practice: "some complex reasoning problems may have relatively simple and easily guessable answers, leading to false positive verification", so they exclude multiple-choice, true/false and proof-based questions, and for general QA they prompt a model "to guess potential answers without any CoT reasoning steps" and drop the prompt if it succeeds "within N attempts"; "N = 8 can remove the majority" of easy-to-hack prompts7 (§2.1). Qwen3's cold-start filter drops queries Qwen2.5-72B-Instruct answers correctly without CoT8 (§4.1). R1 keeps its multiple-choice STEM data, Kimi excludes the format outright, and Qwen3 filters by guessability; none of the three reports ablates the choice.
DAPO's integer-answer rewrite attacks the same problem from the other side: an integer answer space is large enough that a guess is rarely right and simple enough that a parser rarely fails2 (§3.5).
5.5 Curricula and prioritized sampling
Kimi k1.5 uses the pass rate twice. Before RL it ranks difficulty by the pass rate of an SFT model over ten samples at high temperature (lower is harder), which "allows difficulty evaluation to be aligned with the model's intrinsic capabilities" and prefilters trivial cases. During RL it tracks each prompt's success rate \(s_i\) and samples prompts with probability proportional to \(1-s_i\), after a curriculum that starts on easier tasks because "spending a restricted computation budget on very hard problems often yields few correct samples"7 (§2.1, §2.3.4). In its ablation, a warm-up on the full mix followed by hard questions only beat uniform sampling, with the gap shown graphically (Figure 9). Reasoning Gym reports the same direction for its procedurally generated tasks: adaptive difficulty progression outperformed fixed difficulty17 (§1). What none of these reports isolates is the effect of the verifier's error rate on the filter: a prompt whose reference answer is wrong has a pass rate near zero, and every rule above reads that as "too hard".
6. What R1-Zero showed, and what it did not
6.1 The headline, in two versions
R1-Zero's AIME 2024 pass@1 rose from 15.6% to 77.9% in the expanded v2 report6 (§2.3) and from 15.6% to 71.0% in v118 (§2.2.4). Both give 86.7% with consensus decoding. Version 2 describes a 10,400-step run with the maximum length raised from 32,768 to 65,536 tokens at step 8.2k, and says "both the performance and response length of DeepSeek-R1-Zero exhibit a significant jump at the 8.2k step"6 (§2.1). Version 1 gives no step count (only "thousands of RL steps"), learning rate or length schedule18, so the 6.9-point gap cannot be attributed from the papers. The consensus label is inconsistent too: v1 calls 86.7% cons@64, while in v2 the Figure 1 legend reads cons@16 and Supplementary D.1 defines cons@64618. Cite R1-Zero as "77.9% (v2)" or "71.0% (v1)", never bare.
6.2 The base model and the template decide the starting point
R1's own findings section states the dependence on capacity: its 7B dense and 16B MoE bases "consistently failed to yield meaningful improvements" on AIME, and only the 32B dense, 230B MoE and 671B MoE bases gave "substantial performance gains attributable to pure RL training"6 (Supplementary G.1). Dr. GRPO complicates the size story. A 7B Qwen2.5-Math base goes from 16.7% to 43.3% on AIME 2024 under RL, while a plain Llama-3.2-3B base moves from an average of 3.3 to 6.8 across five math benchmarks; after continued pretraining on FineMath the move is 6.0 to 14.8, and on concatenated question-answer text (NuminaQA) 0.14 to 20.73 (Table 4, §3.4). The authors conclude that "math pretraining would improve the ceiling of RL": domain capability in the base matters at least as much as parameter count.
The template matters as much, and Dr. GRPO's Table 1 is the clearest measurement. It evaluates Qwen2.5-Math bases with greedy decoding and a 3,000-token limit under four prompt styles.
| Base model | 4-shot | R1 template | Qwen-Math template | No template |
|---|---|---|---|---|
| Qwen2.5-Math-1.5B | 19.7 (0.0) | 7.9 (0.0) | 24.2 (20.0) | 33.1 (16.7) |
| Qwen2.5-Math-7B | 23.8 (3.3) | 0.0 (0.0) | 26.5 (16.7) | 38.2 (0.2) |
The R1 template, the one in R1's Table 1, takes the 7B model's average to zero, and no template is best, by about 60% over 4-shot prompting. RL then repairs the damage: from Qwen2.5-Math-1.5B, "Templates determine the performance of the initial policies, but RL can improve all policies to a comparable performance of ~40% (given a proper question set)", and with the Qwen-Math template the best final performance came from RL on GSM8K, "much simpler (and o.o.d.) questions", which nearly doubled accuracy on harder ones3 (§3.3, Figure 6). Their reading: "Applying templates in fact destroys the capability before RL reconstructs it. This implies that we should be more conservative in claiming the huge gains brought about by pure RL." Olmo 3 sees the same from the other side: from a base mid-trained without special formatting, "simple" prompt templates "greatly outperform standard post-trained templates (e.g., <think></think>)"9 (§6.1). A format term that rewards a template the base handles badly turns early RL into format recovery, and the headline gain includes it.
6.3 The "aha moment": a narrower conflict than it looks
R1 reports that R1-Zero "exhibits an 'aha moment'... characterized by a sudden increase in the use of the word 'wait' during reflections" and presents reflection as emerging from RL6 (§2.3, Table 2). Dr. GRPO hosted DeepSeek-V3-Base itself, ran the R1 template over 500 MATH questions and found that "DeepSeek-V3-Base also generates a decent amount of self-reflections", including outputs with "Aha" and "wait"; in R1-Zero, self-reflection is more frequent but "nearly half responses with self-reflection do not achieve higher accuracy than those without"3 (§2.3, Appendix E-F). What RL clearly changes is length and frequency: average response length in characters goes from 880.7 to 7,870.3 for unformatted responses, 621.3 to 4,965.4 for correct ones and 1,038.9 to 8,206.1 for incorrect ones, base to R1-Zero3 (Table 5). Neither paper shows that the behavior is new.
R1's own text is closer to Dr. GRPO than the "aha" framing suggests. Its conclusion says "pre-trained checkpoints inherently possess substantial potential for complex reasoning tasks"6 (§6), and Supplementary A.1 says the pretraining data holds "a substantial amount of mathematical and code-related content", which "equips the model with the capability to generate plausible solution candidates, from which reinforcement learning can effectively identify and optimize high-quality outputs". That is selection among candidates the base can already produce. The live disagreement is how much RL adds beyond selection, which Module 20 treats with pass@k.
6.4 Its own problems, and the contamination control you should copy
R1-Zero "faces challenges such as poor readability and language mixing", and its rule-based RL stage "is narrowly focused on reasoning tasks, resulting in limited performance in broader areas such as writing and open-domain question answering"6 (§1); the R1 pipeline (Module 27) was built to fix both.
The subtler problem is whether the benchmark gains measure reasoning. R1 states that its math SFT data and RL prompts were "sourced exclusively from pre-2023 competitions" and n-gram filtered against evaluation sets6 (Supplementary D.1), but the base model is a separate matter. Shao et al. found that on Qwen2.5-Math-7B, RLVR with randomly assigned rewards improved MATH-500 by 21.4 points against 29.1 for ground-truth rewards, with minimal or negative effects on other model families, and attributed it to a clipping bias in GRPO that amplifies high-prior pretraining behaviors19 (abstract, §1). Olmo 3 draws the operational conclusion: open-weight bases with undisclosed mid-training data can be contaminated "which makes spurious rewards as effective as true reward", so it trains RL-Zero from a fully open, decontaminated base with a negative control, random binary rewards, which "does not improve performance on any of our benchmark evaluations"9 (§6, Figure 27).
7. What a binary verdict cannot see
Knowing versus guessing. Under 0-1 grading, Kalai et al. show, "abstaining is strictly sub-optimal": an "I don't know" earns the minimum and an overconfident best guess is optimal20 (§4). A binary verifier trains that incentive into the policy. Damani et al. add a bounded proper scoring rule to the verdict, \(R=\mathbf 1[y=y^*]-(q-\mathbf 1[y=y^*])^2\) with \(q\) the verbalized confidence, and report, against an RLVR model of matched task accuracy, expected calibration error falling from 0.37 to 0.03 on HotpotQA and from 0.26 to 0.10 on math, while RLVR "substantially worsens calibration generalization to new domains"21 (§1, Eq. 1). Everything in Sections 3 to 5 inherits this blind spot: a verifier reads the answer, not the model's confidence in it. Module 24 derives the scoring-rule argument and Module 25 separates the published calibration methods from the name RLCD.
Right answer, wrong reasoning. An outcome verifier cannot tell a derivation from a lucky guess or a memorized answer, which is why Section 5.4 filters guessable prompts rather than fixing the reward. Process supervision is the usual remedy, and R1 reports it did not work for them: steps are hard to define and to judge, and a model-based PRM "inevitably leads to reward hacking"6 (Supplementary G.2). Module 4 (Section 4) has the outcome-versus-process comparison.
Where experts disagree
Two arguments run through the sources: whether reasoning RL should rest on programs alone or may put a model inside the reward, anchored to a reference (Module 16 treats this at length); and whether R1-Zero shows that RL creates reasoning or selects what pretraining already put in the base (Module 20).
A. Programs only for reasoning rewards
DeepSeek-R1 (§2.2)6; DAPO (§2.4)2; Qwen3's endorsement of rule-based rewards (§4.4)8; Tulu 3's verifier-only versus verifier-plus-RM result (§6.2.1)1
Neural reward models are "susceptible to reward hacking during large-scale reinforcement learning" and cost compute to retrain; well-designed rules "assess the correctness of model outputs with high precision, preventing issues like reward hacking".
- R1 saw the hack in its own stage 2: reward rose while Codeforces performance fell.
- A program's errors can be listed and tested, as Section 3.1 does for a regex.
- Coverage stops at checkable answers; R1 calls tasks without a reliable verifier "an open challenge".
- Strict matchers can underestimate by up to 40 points (Math-Verify's authors)12.
B. A model inside the verifier, anchored to a reference or rubric
Kimi k1.5's chain-of-thought equivalence checker7; Qwen3's reference-conditioned judge (§4.4)8; DeepSeek-V3.2's rubric reward model for general tasks (§3)11; Kimi K2's hybrid verification (§3.2.1)14
Free-form answers and equivalent forms defeat regexes; a judge that sees the reference avoids "false negatives that can occur with purely rule-based rewards".
- The one controlled measurement favors it: 98.5 versus 84.4 spot-check accuracy, CoT versus classic model, at math equivalence7.
- The judge has its own error rate and can be exploited; K2 adds a hack-check layer.
- R1 reports limited generalizability of LLM evaluation to open-ended tasks.
C. R1-Zero shows RL creates reasoning behavior
DeepSeek-R1 (§2.3, §6)6
"Sophisticated reasoning behaviors, such as self-verification and reflection, appeared to emerge organically during the reinforcement learning process"; the ingredients are "hard reasoning questions, a reliable verifier, and sufficient computational resources".
- AIME 2024 from 15.6% to 77.9% (v2) with growing length and rising reflective words6.
- On a clean open base, Olmo 3 RL-Zero improves with real rewards and stays flat with random ones9.
- The "aha" is not isolated from the base model's own behavior, and the report's appendix describes selection among the base's candidates.
D. The base model and template supply most of it
Liu et al., Dr. GRPO3; Shao et al., Spurious Rewards19; Yue et al. on pass@k22; Olmo 3's contamination argument9
DeepSeek-V3-Base already reflects; applying templates "destroys the capability before RL reconstructs it"; random rewards lift Qwen2.5-Math-7B by 21.4 points on MATH-500 against 29.1 for ground truth19; at large \(k\) base models can beat their RLVR versions22.
- Replicable measurements (Table 7; random-reward runs) and a practical control.
- Spurious-reward gains are model-specific; minimal or negative on other families.
- Olmo 3's flat random-reward control means contamination does not explain its gains.
- Large-\(k\) pass@k results are contested (Module 20).
A versus B. Use the cheapest verifier whose error rate you have measured on held-out policy samples, not on reference solutions. Where a canonical form exists (integers, SQL result sets, unit tests, constraint checkers) use a program, because its failure modes can be listed and fixed. Where a strict matcher's false-negative rate is material, anchor a model to the reference, as k1.5 did, and keep every program-level check that still applies. Nobody has run the decisive experiment: identical prompts and RL settings under a rule-only reward, a reference-anchored judge and a preference model, audited by humans for false positives and negatives. Tulu 3's Figure 22 is the nearest, a single GSM8K setup. With no reference at all, A has nothing to offer and B becomes the judge question of Module 16.
C versus D. Both are partly right, and less opposed than their titles. RL demonstrably changes response length and the frequency of reflective language, and on a decontaminated base it demonstrably produces gains (Olmo 3's null control). The base and the template set the starting point and an unknown share of the gain: the R1 template takes Qwen2.5-Math-7B to an average of 0.0 in Dr. GRPO's Table 1, and R1's appendix itself describes selection among the base's candidates. Large-\(k\) pass@k on a base with fully known pretraining data, which Olmo 3 RL-Zero was built to allow, would settle it. Until then, compare base and RL under the same template, state which R1 version you cite, and run the random-reward control.
What is changing
Verifier scope is widening at both ends. At the formal end, NeMo Gym lists Lean 4 proof-verification environments and a proof-verification environment scored against ground truth and meta-verifier agreement16, and DeepSeek-V3.2-Speciale takes its proof reward from DeepSeekMath-V211. At the soft end, 2026 recipes train generative reward models inside the loop (Module 16). Environments, not datasets, are becoming the unit of verifier design: Nemotron 3 Ultra runs one unified RLVR stage across terminal, office, software-engineering, search, math, code and chat environments, varies harnesses to avoid overfitting to one, and "perform[s] reward profiling prior to training"23 (§3.2), which is Section 5's prompt-selection problem moved into the environment catalogue. Prompt selection is shifting from offline filters to online ones (Module 11). Olmo 3 is the only report read for this module that runs a random-reward negative control. Binary correctness is being supplemented by calibration-aware terms (Module 24)21. Treat the tables as dated 2026-10-27: NeMo Gym is a moving catalogue (the page shows v0.7.0), TRL documents its main branch, and R1's AIME figure has already changed once.
References
- paperLambert, N., Morrison, J., Pyatkin, V., Huang, S. et al. (Allen Institute for AI). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124 v5 (14 Apr 2025); v1 22 Nov 2024. https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: RLVR definition and Eq. 7-8 (§6), alpha = 10, the verifiable prompt mix (§6.1, Table 22), implementation details (§6.2), the RM-score-on-top ablation (Fig. 22), IFEval over-optimization samples (Appendix B.4, Fig. 28), final-model context (Table 23).
- paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y. et al. (ByteDance Seed, Tsinghua AIR). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476 v2 (20 May 2025). https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Used for: §2.3 (KL removed), §2.4 and Eq. 7 (+1/-1 rule-based reward), §3.2 and Fig. 3b (accuracy-1 prompts accumulate), §3.5 (integer-answer transformation, DAPO-Math-17K).
- paperLiu, Z., Chen, C., Li, W., Qi, P. et al. (Sea AI Lab, NUS, SMU). Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv:2503.20783 v2 (6 Oct 2025). https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Used for: §2.1-§2.3 (templates, Table 1, self-reflection in DeepSeek-V3-Base), §3.2 (binary reward with Math-Verify), §3.3 and Fig. 6 (template and question-set coverage), Table 4, Appendix F-G.
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv:2504.12501 v12 (11 Sep 2026). https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Used for: p. 14 (R1 “popularizing RLVR”); Ch. 7 pp. 97-99 (RLVR definition, the RLGT naming history, unit-test reward variants) and pp. 105-106 (common practices: difficulty filtering at 20-80%, format, language-consistency and length rewards).
- paperShao, Z., Wang, P., Zhu, Q., Xu, R. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 v3 (27 Apr 2024). https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: §4.2 (reward model trained from rule-judged labels; DeepSeekMath-RL setup) and §5.2.1 (‘Rule’ versus ‘Model’ reward functions).
- paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 v2 (4 Jan 2026). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: §2.1 (GRPO settings), Table 1 (template), §2.2 and Eq. 4 (rewards), §2.3 (15.6% to 77.9%, aha moment), §3.2.1 and Eq. 7 (language consistency), Appendix A.1 (base-model data), B.1 (rule-based reward module), B.3.1-B.3.2 and Table 4 (RL data, test generation), B.5, B.6, D.1 (decontamination, cons@64), G.1-G.2, §6 (limitations).
- paperKimi Team (Moonshot AI). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599 v4 (3 Jun 2025); v1 22 Jan 2025. https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Used for: §2.1 prompt-set curation (difficulty from ten samples, exclusion of multiple-choice, true/false and proof questions, guess-without-CoT filter with N = 8), §2.3.4 sampling strategies, appendix test-case generation (1,000 to 614 to 463 to 323) and math reward modeling (84.4 versus 98.5 spot-check accuracy).
- paperYang, A., Li, A., Yang, B., Zhang, B. et al. (Qwen Team, Alibaba). Qwen3 Technical Report. arXiv:2505.09388 v1 (14 May 2025). https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Used for: §4.1 (query filtering), §4.2 (3,995 query-verifier pairs, four criteria, AIME’24 70.1 to 85.1 in 170 steps), §4.4 (three reward types in General RL).
- paperTeam Olmo (Allen Institute for AI). Olmo 3. arXiv:2512.13961 v2 (14 Apr 2026); v1 15 Dec 2025. https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: §4.4.1 (OlmoRL verifiers for math, code, instruction following, chat; Figure 16), §4.4.2 and Table 20 (Dolci-Think-RL, offline difficulty filter at 62.5%), §4.4.3 (inference cost), §4.5 (mixed-domain RL), §6 (RL-Zero data, templates, random-reward control, Fig. 27).
- paperZhou, J., Lu, T., Mishra, S., Brahma, S. et al. (Google, Yale). Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911 v1 (14 Nov 2023). https://arxiv.org/pdf/2311.07911. Accessed 2026-10-27. Used for: 25 verifiable instruction types and 541 prompts (§1), the strict versus loose metric and the markdown false-negative example (§2.2). Not in the researcher’s source map; fetched for this module.
- paperDeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556 v1 (2 Dec 2025). https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: §3 Mixed RL Training: rule-based outcome reward, length penalty and language-consistency reward for reasoning and agent tasks; rubric-based generative reward model for general tasks; DeepSeekMath-V2 proof reward in Speciale.
- repoHugging Face. Math-Verify (README). GitHub, main branch. https://raw.githubusercontent.com/huggingface/Math-Verify/main/README.md. Accessed 2026-10-27. Used for: Three-step grading architecture (answer extraction, SymPy conversion, gold comparison) and the claim that strict evaluators can underestimate model performance by up to 40 points. Not in the researcher’s source map; fetched for this module.
- officialHugging Face. TRL documentation: GRPO Trainer (main branch). docs page, unversioned snapshot. https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: Custom reward-function interface, the format-check and boxed exact-match examples (Examples 2 and 3), reward summing with reward_weights, multi_objective_aggregation (GDPO note), the scale_rewards caveat, the frac_reward_zero_std metric.
- paperKimi Team (Moonshot AI). Kimi K2: Open Agentic Intelligence. arXiv:2507.20534 v2 (3 Feb 2026); v1 28 Jul 2025. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Used for: §3.2.1 Verifiable Rewards Gym: moderate difficulty from SFT-model pass@k, hybrid rule and LLM verification with a hack-check layer, sandbox scale.
- paperMacDiarmid, M., Wright, B., Uesato, J., Benton, J. et al. (Anthropic, Redwood Research). Natural Emergent Misalignment from Reward Hacking in Production RL. arXiv:2511.18397 v1 (23 Nov 2025). https://arxiv.org/pdf/2511.18397v1. Accessed 2026-10-26. Used for: §3: the three named coding-environment hacks (AlwaysEqual object, sys.exit(0) before asserts, conftest.py report patching).
- repoNVIDIA. NeMo Gym (README; the page shows release v0.7.0). GitHub, main branch. https://raw.githubusercontent.com/NVIDIA-NeMo/Gym/main/README.md. Accessed 2026-10-27. Used for: Environment definition (dataset, agent harness, verifier, state) and verifier rows: Bird SQL (1,534 tasks, unordered result-set equality), Spider2 Lite (135 tasks), Blackjack (+1/0/-1), GUI coordinate (smooth quadratic), single-step tool use (exact match), Lean 4 and proof-verification environments.
- paperStojanovski, Z., Stanley, O., Sharratt, J., Jones, R. et al. Reasoning Gym: Reasoning Environments for Reinforcement Learning with Verifiable Rewards. arXiv:2505.24760 v2 (20 Oct 2025). https://arxiv.org/pdf/2505.24760v2. Accessed 2026-10-26. Used for: Abstract and §2 design principles P1 (algorithmic verifiability), P2 (large solution spaces), P3 (parametric difficulty control).
- paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (version 1). arXiv:2501.12948 v1 (22 Jan 2025). https://arxiv.org/pdf/2501.12948v1. Accessed 2026-10-26. Used for: §2.2.2-§2.2.4: 15.6% to 71.0% pass@1 and 86.7% with majority voting (cons@64); the original reward description; absence of training hyperparameters.
- paperShao, R., Li, S. S., Xin, R., Geng, S. et al. Spurious Rewards: Rethinking Training Signals in RLVR. arXiv:2506.10947 v2 (25 Feb 2026). https://arxiv.org/pdf/2506.10947v2. Accessed 2026-10-26. Used for: Abstract and §1: random rewards improve MATH-500 by 21.4 points on Qwen2.5-Math-7B versus 29.1 for ground truth; effect minimal or negative on other families.
- paperKalai, A. T., Nachum, O., Vempala, S. S., Zhang, E. (OpenAI, Georgia Tech). Why Language Models Hallucinate. arXiv:2509.04664 v1 (4 Sep 2025). https://arxiv.org/pdf/2509.04664v1. Accessed 2026-10-26. Used for: §4: under binary 0-1 grading, abstaining is strictly suboptimal and a best guess is optimal.
- paperDamani, M., Puri, I., Slocum, S., Shenfeld, I. et al. (MIT). Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (RLCR). ICLR 2026; arXiv:2507.16806 v2 (15 May 2026). https://arxiv.org/pdf/2507.16806v2. Accessed 2026-10-26. Used for: Abstract and §1, Eq. 1: binary correctness plus Brier score of verbalized confidence; expected calibration error 0.37 to 0.03 (HotpotQA) and 0.26 to 0.10 (math).
- paperYue, Y., Chen, Z., Lu, R., Zhao, A. et al. (Tsinghua LeapLab, SJTU). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?. arXiv:2504.13837 v5 (24 Nov 2025). https://arxiv.org/pdf/2504.13837v5. Accessed 2026-10-26. Used for: Abstract and Fig. 1: RLVR models win at small k, base models win at large k; coverage (pass@256) can fall while pass@1 rises.
- paperNVIDIA. Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv:2606.15007 v1 (12 Jun 2026). https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Used for: §3.2: one unified RLVR stage across all environments, reward profiling before training, harness diversity.