Reinforcement Learning for LLM Post-Training · Module 29
Case Study: Qwen3: Four-Stage Reasoning Pipeline, 3,995 Query-Verifier Pairs, Thinking-Mode Fusion, and Strong-to-Weak Distillation
Qwen3 (April and May 2025) documents its post-training in about three pages: a deliberately small cold start, one reasoning-RL stage on 3,995 query-verifier pairs, a fusion step that puts thinking and non-thinking behavior into one set of weights, a general-RL stage over more than 20 tasks, and, for six smaller models, distillation in place of all four. This module reads each stage with the report's own numbers, sets the pipeline beside R1's, and lists what the report does not print. Afterwards you can say why Qwen3 distilled its small models, how small its RL set is, what the later stages cost on hard reasoning benchmarks, and which claims about the recipe rest on a table and which on a sentence.
1. The recipe in one page: two routes, four stages
1.1 Two objectives, two routes
The report's post-training section opens with two goals. The first is thinking control: a "non-thinking" and a "thinking" mode in one model, with a user-set token budget for the thinking. The second is strong-to-weak distillation: use the large models to cut the cost of building the small ones1. The flagship models, Qwen3-235B-A22B (235B parameters, 22B activated per token) and the dense Qwen3-32B, "follow a sophisticated four-stage training process". The lightweight models, dense 0.6B, 1.7B, 4B, 8B and 14B plus the 30B-A3B MoE, skip those stages and are distilled from the flagships1 (§1, §4, §4.5, Figure 1). The first two stages build the thinking ability; the last two add the non-thinking one1.
Three dates frame how to read it. Qwen's release post is dated 29 April 20252. The technical report went to arXiv on 14 May 2025, and the arXiv listing shows no later version as of 2026-10-271. The report therefore describes the April 2025 models only; the last section of this module covers what Qwen shipped afterwards.
1.2 Where it comes from
The nearest ancestor is Qwen's own QwQ-32B, announced on 6 March 2025. Its blog describes a two-stage RL run from a cold-start checkpoint. The first stage covers math and code only, with an accuracy verifier for math answers and a code execution server that runs predefined test cases, and no learned reward model. The second stage targets general capabilities with "rewards from general reward model and some rule-based verifiers", and the post says it used "a small amount of steps"3. Qwen3's stage 2 and stage 4 have the same shape, a verifier-only reasoning stage followed by a broader stage that admits learned rewards. That lineage is my reading; the Qwen3 report does not claim it. The documented link is more direct: QwQ-32B generates the candidate responses for Qwen3's cold-start data and is the main RL-trained baseline in the 32B-class tables1 (§4.1, Tables 13 and 15).
1.3 What the report offers as evidence
The training description is about three pages (§4.1 to §4.5, pp. 9 to 12); the rest of §4 is evaluation1. Four results carry everything this module can say quantitatively:
- One sentence on the flagship's reasoning RL: AIME'24 rises from 70.1 to 85.1 over 170 steps (§4.2).
- Table 22, which follows Qwen3-32B through Stages 2, 3 and 4 on 13 benchmarks (§4.7).
- Table 21, which compares RL with on-policy distillation on Qwen3-8B from the same starting checkpoint (§4.7).
- Tables 11 to 20, which score the final models against baselines (§4.6).
There are no RL training curves, no per-stage table for the 235B flagship, and no ablation of the 3,995-pair choice, the cold-start size or any reward type. Where a mechanism has its own module, this one points to it instead of re-deriving it: GRPO is Module 9, reward routing and judges Module 16, on-policy distillation and the cost arithmetic of Table 21 Module 21, the chat-template and budget mechanics of fusion Module 23.
2. Stage 1: a long-CoT cold start that is a filter, not a training run
The cold-start stage is described almost entirely as data selection. The dataset spans math, code, logical reasoning and general STEM, and every problem is paired with verified reference answers or code-based test cases1 (§4.1). Two filters build it.
Query filtering. Qwen2.5-72B-Instruct removes queries that are not easily verifiable, namely those with several sub-questions or those asking for general text generation. It also removes queries it can answer correctly without chain-of-thought reasoning, which the report says "helps prevent the model from relying on superficial guessing". It labels each query's domain so the mix stays balanced, and a validation query set is held out before any generation1 (§4.1).
Response filtering. QwQ-32B generates \(N\) candidate responses per remaining query. When QwQ-32B "consistently fails", human annotators assess the responses. For queries with a positive pass@\(N\), six defects are filtered out: wrong final answers, substantial repetition, guesswork without adequate reasoning, inconsistency between the thinking and the summary, inappropriate language mixing or stylistic shifts, and suspected overlap with validation items1 (§4.1).
The stated purpose explains the stage's size policy. The objective is "to instill foundational reasoning patterns in the model without overly emphasizing immediate reasoning performance", so that the model's potential "is not limited" before RL; the report therefore prefers to "minimize both the number of training samples and the training steps"1 (§4.1). The stage is under-trained on purpose. It is the format-and-style prior that Stage 2 then optimizes, and the report gives neither \(N\), nor the number of examples, nor the step count, nor the learning rate.
Two comparisons with R1 are worth holding. R1 also cold-starts, with "thousands" of long chain-of-thought examples, and gives a product reason: it wanted the reasoning to read in the first person, to begin by comprehending the problem, and to keep one language4 (§3, Supp. B.3.2). Qwen3 handles the same symptom, language mixing, on the data side: item five of the response filter removes it from the cold-start traces. R1 additionally added a language-consistency term to the RL reward, which its own ablation says costs a slight performance drop4 (§3.2.1). Second, the 235B-A22B flagship's cold start imitates a 32B dense predecessor. The stage-1 description names no other generator, so the first stage of the largest model is a distillation step from a smaller reasoning specialist (my reading of §4.1).
3. Stage 2: reasoning RL on 3,995 query-verifier pairs
3.1 Four criteria, one number
The RL set has to satisfy four criteria. The pairs were not used in the cold-start phase; they are learnable for the cold-start model; they are as challenging as possible; and they cover a broad range of sub-domains. The paragraph ends: "We ultimately collect a total of 3,995 query-verifier pairs"1 (§4.2). Figure 2 draws the data gates of both stages. The end count is the only number the report prints for either: the pool before selection, the cold-start subset size, and any pass-rate threshold behind "learnable" and "as challenging as possible" are not given.
Read together, criteria 2 and 3 bound the same band that Module 10 describes for prompt selection and that Module 11 explains through zero-variance groups: a prompt the cold-start model never solves, or always solves, gives every rollout in a group the same reward and so contributes no group-relative gradient. This is the writer's reading; the report states the criteria without a rationale.
The scale is the notable part. R1's reasoning RL data is 26K math, 17K code, 22K STEM and 15K logic prompts, 80K in all, plus 66K general prompts4 (Supp. Table 4); the text of Supp. B.3.1 adds 8K bug-fixing problems to the 17K algorithm questions, which would make 88K. \(80{,}000 / 3{,}995 = 20.0\) and \(88{,}000 / 3{,}995 = 22.0\), so the R1 set is about twenty times larger (derived; R1's counts are rounded to thousands). Qwen3 chose curation over volume, with the criteria above as the only filter on record.
3.2 What the report says about the optimizer
Four statements, all from §4.21:
- The algorithm is GRPO, cited to Shao et al. (2024)5; Module 9 derives it.
- The authors observe that "using a large batch size and a high number of rollouts per query, along with off-policy training to improve sample efficiency, is beneficial to the training process."
- The authors "addressed how to balance exploration and exploitation by controlling the model's entropy to increase steadily or remain stable". That is a target; Module 13 notes that no method for reaching it is given.
- The result is "consistent improvements in both training reward and validation performance over the course of a single RL run, without any manual intervention on hyperparameters".
No batch size, rollout count, degree of off-policyness, learning rate, KL coefficient, clip range or maximum length is printed. One later paper, ScaleRL, reads more into the report than it says, twice. It states that the PPO-off-policy-\(k\) scheme "has been used previously by Qwen3"6 (§3.1); the Qwen3 text says only "off-policy training to improve sample efficiency" and defines no scheme or \(k\). ScaleRL's appendix also lists Qwen3 among works that use interruptions to stop over-long generations during RL6 (App. A.10); the only interruption in the Qwen3 report is the stop-thinking string appended at inference time, which the report says "is not explicitly trained"1 (§4.3). I could not find an RL-time interruption in the report text. Treat both attributions as second-hand readings.
3.3 The one curve, and how many passes it implies
The report's only flagship RL measurement is one sentence: "the AIME'24 score of the Qwen3-235B-A22B model increases from 70.1 to 85.1 over a total of 170 RL training steps"1 (§4.2). That is \(85.1-70.1 = 15.0\) points, 0.088 points per step on average (derived). The start is not labelled; by its place in the pipeline it is the cold-start checkpoint. A "step" is not defined, and with several minibatches per rollout the number of rollout batches and the number of optimizer updates differ by that factor (Module 1).
The number of unknowns matters because "3,995 pairs" is only small if the batches are. With \(P\) prompts per step and \(S\) steps, the run makes \(P\cdot S\) prompt draws and \(P\cdot S/3{,}995\) passes over the set. R1-Zero drew 32 unique questions per step4 (§2.1); at \(P=32\), 170 steps is 5,440 draws, 1.36 passes. At \(P=128\) it is 21,760 draws, 5.45 passes. The report does not say which regime it is in, and so does not say whether the set is seen about once or more than five times.
Passes over the 3,995-pair set in a short reasoning-RL run
Formulas: \(P S\), \(P S / 3{,}995\), \(3{,}995 / P\), \(P S G\). Qwen3 reports none of \(P\), \(G\) or the meaning of a step; the defaults are illustrative. R1-Zero's values were \(P=32\), \(G=16\), \(S=10{,}400\)4.
One more reading error to avoid. The final Qwen3-235B-A22B scores 85.7 on AIME'24 in thinking mode1 (Table 11), against 85.1 after reasoning RL. The two numbers sit in different parts of the report, and the report does not say that 85.1 was scored under Table 11's protocol (64 samples per question, 38,912 maximum output tokens for AIME). A +0.6 difference is therefore not evidence that Stages 3 and 4 were free at 235B scale. The only stage-wise measurement is the 32B model's, where AIME'24 goes from 83.8 to 81.4 (Section 5.4).
4. Stage 3: thinking-mode fusion, read for its data
The mechanics of fusion, meaning the /think and /no think flags, the empty think block kept in non-thinking samples, the rule that in a multi-turn chat the model follows the last flag, and the stop-thinking string that implements the budget, are Module 23 §5's subject. This section stays with what the stage does to the pipeline's data.
Fusion is "continual supervised fine-tuning (SFT) on the Reasoning RL model"1 (§4.3). Its dataset has two halves with different origins. The thinking half is produced by rejection sampling on Stage 1 queries "using the Stage 2 model itself", so that the Stage 2 model's performance "is not compromised by the additional SFT". The non-thinking half is "carefully curated" across coding, mathematics, instruction following, multilingual tasks, creative writing, question answering and role-playing, scored with automatically generated checklists, with the share of translation tasks raised to help low-resource languages. The report gives no sample counts, no thinking-to-non-thinking ratio and no epochs.
Three readings follow, all the writer's. First, the thinking half is the policy's own filtered output, so it asks the model to change little in thinking behavior; the non-thinking half carries the new behavior. That is a regularizer by construction, and Table 22 measures how well it works: in thinking mode this stage alone moves AIME'24 from 83.8 to 81.9 and LiveCodeBench v5 from 68.4 to 67.21 (Table 22, printed deltas of -1.9 and -1.2). Second, fusion puts an off-policy step between the two RL stages, so Stage 4 starts from an SFT checkpoint and not from the RL one; the report does not discuss what that does to the entropy that Stage 2 managed. Third, the checklist grading of non-thinking data is the report's only rubric-style filter; Module 16 covers rubrics as rewards, and here they select SFT data.
The budget ability is described as "not explicitly trained but emerges naturally as a result of applying Thinking Mode Fusion"1 (§4.3). Module 23 weighs that claim against designs that train the budget.
5. Stage 4: general RL over more than 20 tasks
5.1 Capabilities, tasks and where the stage sits
The stage "aims to broadly enhance the models' capabilities and stability across diverse scenarios", with "a sophisticated reward system covering over 20 distinct tasks, each with customized scoring criteria"1 (§4.4). The report lists five target capability groups and not the tasks themselves:
- Instruction following: content, format, length and structured-output requirements.
- Format following: responding to the
/thinkand/no thinkflags by switching modes, and using the<think>and</think>tokens consistently. This is the RL stage that polishes the mode switching fusion installed (Module 23 gives the ThinkFollow numbers). - Preference alignment: helpfulness, engagement and style on open-ended queries.
- Agent ability: invoking tools through designated interfaces, trained with multi-turn rollouts against real environment feedback.
- Specialized scenarios: for example retrieval-augmented generation, where reward signals aim to reduce hallucination.
The MOPD paper files this report under "mixed RL": samples from different domains in one training batch, each still scored by its own per-domain reward and advantage8 (§1). Qwen3 is both things at once, a cascade across stages (reasoning RL, then an SFT step, then general RL) and a mixture inside Stage 4. Module 22 compares mixed RL, cascades and multi-teacher distillation; the Qwen3 report itself offers no comparison among them.
5.2 Three reward types, and how little of the map is printed
The report names three reward types1 (§4.4), each with the reason it gives:
- Rule-based. Described as "also useful for general tasks such as instruction following (Lambert et al., 2024) and format adherence"; "well-designed rule-based rewards can assess the correctness of model outputs with high precision, preventing issues like reward hacking". The citation is Tulu 3, whose RLVR stage verifies IFEval-style constraints9 (§1).
- Model-based, with a reference answer. Qwen2.5-72B-Instruct scores the response against the reference. The stated benefit is "more flexible handling of diverse tasks without requiring strict formatting, avoiding false negatives that can occur with purely rule-based rewards".
- Model-based, without a reference. A reward model trained on human preference data assigns scalar scores. It "does not depend on a reference answer", handles "a broader range of queries" and enhances "engagement and helpfulness".
The matrix is sparse because the report is. It does not say which of the 20-plus tasks uses which type, how a 0/1 rule reward, a judge score on an unstated scale and an RM scalar are combined or normalized, what the RM's size or training data are, what the judge prompt looks like, or how many steps the stage ran. Module 16 covers what such a router has to settle (the judge's failure modes, reference-conditioned scoring and rubric alternatives), and Module 19 the hacking that a scalar RM invites.
R1's authors reach the first two types independently: rule-based reward models and LLMs that judge correctness "against a predefined ground-truth" are, in their words, "robust mechanisms for mitigating issues related to reward hacking", with the LLM route limited to well-defined, concise answers4 (Supp. G.1). For open-ended data R1 used reward models too, and capped their exposure: preference reward entered only in the last 400 of 1,700 second-stage steps, because longer exposure "may lead to reward hacking"4 (§3.2.2). Qwen3 prints no equivalent exposure figure, no hacking observation and no cap on the preference RM (type 3). It also scores Stage 4 with a fixed, previous-generation judge, Qwen2.5-72B-Instruct, against policies up to 235B parameters; whether that judge's ceiling bound the stage is not discussed.
5.3 Agent training: one sentence of mechanism
The report's whole description of agent RL is one sentence: "During the RL rollout, the model is allowed to perform complete multi-turn interaction cycles with real environment execution feedback, thereby improving its performance and stability in long-horizon decision-making tasks"1 (§4.4). It names no environment, tool set, turn cap, reward definition or sandbox. Module 17 lists what an environment contract has to specify, and Module 18 how multi-turn credit assignment works; neither can be filled in from this report.
The measurements are in Table 22. The public tool-calling benchmark BFCL v3 moves from 68.4 to 70.3 in thinking mode at Stage 4 (+1.9) and from 61.5 to 63.0 in non-thinking mode (+1.5). The in-house ToolUse benchmark, which scores "intent recognition, format accuracy, and parameter accuracy" across single-turn, multi-turn and multi-step calls, moves from 70.4 to 85.5 (+15.1) and from 73.2 to 86.5 (+13.3)1 (Table 22). The two largest Stage 4 gains in thinking mode, ToolUse (+15.1) and ThinkFollow (+10.2), are both on in-house benchmarks the report built to monitor the capabilities that stage trains. That does not show overfitting; the in-house sets measure format and stability behaviors BFCL may not. It does mean the biggest agent number has no external check.
5.4 What Stages 3 and 4 bought, and what they cost
Table 22 is the report's only stage-by-stage measurement, and it is for the dense 32B model, not the flagship. Figure 4 redraws its thinking-mode columns as the change at each stage and the total from Stage 2 to Stage 4.
Four things follow. The scores and deltas are the report's; the sums, shares and counts below are derived from them.
- The gains are in what the stages train. From Stage 2 to Stage 4: ToolUse +22.2, CounterFactQA +17.7, IFEval +12.0, Multi-IF +11.6, LengthCtrl +10.9, Arena-Hard +7.0, LiveBench +6.3. The public agent benchmark BFCL v3 gains +1.3.
- The costs are on the hardest reasoning benchmarks. AIME'24 falls 2.4 points (83.8 to 81.4) and LiveCodeBench v5 2.7 (68.4 to 65.7); MMLU-Redux and GPQA-Diamond move by -0.5 and -0.4. The report conjectures the cause is training on "a broader range of general tasks" and states that it chose to accept the trade-off for versatility1 (§4.7); Module 23 quotes the passage.
- Most of the AIME'24 loss arrives at the SFT step, not the RL step. Stage 3 contributes -1.9 of the -2.4, 79% (derived); for LiveCodeBench v5, -1.2 of -2.7, 44%. The report attributes the drop to "these two training stages" together and offers no split; its own table supports one. With one run per cell, differences under a point should not carry an argument.
- In non-thinking mode, Stage 4 costs almost nothing. Between Stage 3 and Stage 4 eleven of twelve shared rows rise (AIME'24 +2.5, GPQA-Diamond +4.3, ToolUse +13.3) and only MMLU-Redux falls (-1.0)1 (Table 22). The regression is a thinking-mode phenomenon.
For scale on the benchmark itself: AIME has 30 questions per year and the report averages 64 samples per question1 (§4.6), so one question's worth of accuracy is \(100/30 = 3.33\) points. A 2.4-point change is 0.72 of one question's worth (derived), and the report prints no seeds or intervals; Module 20 treats how much such differences can bear.
6. Strong-to-weak distillation: the route for six models
6.1 Two phases, no RL
The distillation pipeline covers five dense models (0.6B, 1.7B, 4B, 8B, 14B) and the 30B-A3B MoE1 (§4.5). It has two phases.
- Off-policy distillation. The outputs of teacher models, generated in both
/thinkand/no thinkmodes, are combined for response distillation. The report says this gives the students "basic reasoning skills and the ability to switch between different modes of thinking". - On-policy distillation. In the report's words, "prompts are sampled, and the student model produces responses in either /think or /no think mode. The student model is then fine-tuned by aligning its logits with those of a teacher model (Qwen3-32B or Qwen3-235B-A22B) to minimize the KL divergence."
The motivation is a result from "preliminary experiments": directly distilling output logits into lightweight students improves performance while "maintaining fine-grained control over their reasoning processes", and it removes the need to run the four-stage process for each small model. The introduction states the finding bluntly: "Distillation from advanced teacher models significantly outperforms reinforcement learning in performance and training efficiency"1 (§1, §4). Module 21 derives the objective. The report does not say which teacher serves which student, the KL direction, the estimator, the number of prompts or steps, or whether the off-policy phase filters teacher outputs for correctness.
One consequence deserves stating. The six students never see an RL reward. Their thinking behavior, non-thinking behavior and mode switching all come from teacher signals, so the Stage 3 and Stage 4 trade-offs of Section 5.4 are not measured for them; the report has no stage-wise table for any small model.
6.2 What the "1/10" figure measures
The report writes a GPU-hour ratio of about one tenth twice, for two different comparisons. In the §4 introduction, distillation needs "only 1/10 of the GPU hours compared to the four-stage training method". In §4.7, on-policy distillation reaches higher scores than direct RL "while requiring approximately only 1/10 of the GPU hours", and Table 21 gives the numbers: starting from the same off-policy-distilled Qwen3-8B checkpoint, on math and code queries only, the RL arm used 17,920 GPU-hours and the on-policy-distillation arm 1,8001 (Table 21, p. 21; the two figures appear only in the table's GPU Hours column, and no sentence of the text states them). Module 21 §4.3 has the arithmetic (a ratio of 9.96 from the printed numbers), the observation that the passage gives no hardware and does not say whether the teacher is included in the 1,800, and the per-point costs. Module 20 covers how pass@64 should be estimated; the report does not say how it did, and the pass@64 gain for on-policy distillation is exactly one question on each AIME set (Module 21).
Three further points concern what the ratio is a ratio of.
- The introduction's 1/10 has no printed support. The report gives no GPU-hours for the four-stage route, for any of its stages, or for the off-policy phase. Only the §4.7 comparison has numbers.
- The §4.7 ratio compares two continuations. Both arms start from the off-policy-distilled checkpoint, so that phase's cost, and the cost of generating teacher outputs for it, sits in neither 17,920 nor 1,800. The table's off-policy row has a dash in the GPU-hours column. It is not a measurement of the whole small-model recipe against the whole flagship recipe.
- Table 21's final row is not the shipped model. The released Qwen3-8B scores 76.0 on AIME'24 and 67.3 on AIME'25 in thinking mode1 (Table 17), against 74.4 and 65.5 for the Table 21 distillation row. Table 21 restricts to "math and code-related queries" for simplicity; the report does not say how the experiment relates to the released checkpoint.
The Thinking Machines post builds on this table. It takes the RL cost from the report and starts from its own SFT baseline of 60% on AIME'24, against 55.0 for the off-policy row here10; Module 21 compares the two accountings.
6.3 What the students score, and where the claim thins
Table 1 compares each student's thinking-mode scores with the R1-distilled baselines the report chose, plus QwQ-32B, on the four benchmarks where Tables 15, 17 and 19 overlap and the claim is testable. Differences are student minus baseline, in points, computed from the printed scores.
| Student | Baseline | Param ratio | AIME'24 | AIME'25 | LCB v5 | GPQA-D |
|---|---|---|---|---|---|---|
| Qwen3-30B-A3B (3B active) | QwQ-32B | 10.7 | +0.9 | +1.4 | -0.1 | +0.2 |
| Qwen3-30B-A3B (3B active) | R1-Distill-Qwen-32B | 10.7 | +7.8 | +21.3 | +8.1 | +3.7 |
| Qwen3-14B | R1-Distill-Qwen-32B | 2.3 | +6.7 | +20.8 | +9.0 | +1.9 |
| Qwen3-8B | R1-Distill-Qwen-32B | 4.0 | +3.4 | +17.7 | +3.0 | -0.1 |
| Qwen3-8B | R1-Distill-Qwen-14B | 1.8 | +6.3 | +22.8 | +12.0 | +2.9 |
| Qwen3-4B | R1-Distill-Qwen-32B | 8.0 | +1.2 | +16.0 | -0.3 | -6.2 |
| Qwen3-4B | R1-Distill-Qwen-14B | 3.5 | +4.1 | +21.1 | +8.7 | -3.2 |
| Qwen3-1.7B | R1-Distill-Llama-8B | 4.7 | -2.1 | +9.0 | -9.3 | -8.9 |
| Qwen3-1.7B | R1-Distill-Qwen-1.5B | 0.9 | +19.4 | +14.0 | +20.0 | +6.3 |
| Qwen3-0.6B | R1-Distill-Qwen-1.5B | 2.5 | -18.2 | -7.7 | -0.9 | -5.9 |
Read the table in three bands. From 4B up, the students lead the R1-distilled baselines on AIME'25 by 16.0 to 22.8 points and on AIME'24 by 1.2 to 7.8, with 4B against R1-Distill-Qwen-32B the tightest (+1.2) and 8 times fewer parameters. On LiveCodeBench v5 and GPQA-Diamond the lead is smaller or absent: the 4B student trails R1-Distill-Qwen-32B by 0.3 and 6.2 points, and Qwen3-8B ties it on GPQA-Diamond (-0.1). At the bottom, the claim thins. Qwen3-1.7B loses to R1-Distill-Llama-8B on three of four benchmarks, and Qwen3-0.6B trails a 1.5B baseline on all four. The report's sentence, that the edge-side models "outperform baselines even with more parameters"1 (§4.6), holds on AIME from 4B up and holds only in part below.
Four cautions bound any attribution to the on-policy phase. First, the baselines are SFT-only models, since R1's authors "apply only SFT and do not include an RL stage"4 (Supp. F), so the comparison is partly SFT-only against SFT plus on-policy distillation; but bases, teachers, data and sizes all differ too. Second, the baselines' AIME'24 and GPQA-Diamond scores in Tables 15, 17 and 19 (AIME'24 28.9, 50.4, 69.7, 72.6; GPQA-Diamond 33.8, 49.0, 59.1, 62.1) equal, to the decimal, those DeepSeek printed for the same models4 (Supp. Table 15), which suggests they were carried over rather than re-run. Their LiveCodeBench values were not: the report's 54.5 for R1-Distill-Qwen-32B against DeepSeek's 57.2, and 45.5 against 53.1 for the 14B, are different measurements on a different window (Qwen's 2024.10 to 2025.02 against DeepSeek's 2024-08 to 2025-01), and DeepSeek's table has no AIME'25 column at all. The Qwen3 report does not say which baseline cells were taken and which were run, so a mixed protocol is possible. Third, the Qwen3 students are scored at 32,768 maximum output tokens (38,912 for AIME), and the baselines' settings are not stated1 (§4.6). Fourth, every cell is one run.
6.4 The parity with QwQ-32B
The first row of Table 1 is the result the report highlights: Qwen3-30B-A3B "achieves comparable performance to QwQ-32B with a smaller model size and less than 1/10 activated parameters"1 (§4.6). QwQ-32B is an RL-trained model3, and the distilled MoE matches it within 1.4 points on all four benchmarks. For decode, the arithmetic is direct: weights read per generated token at batch 1 in BF16 are about \(3\times10^9 \times 2\ \mathrm{B} = 6\) GB for 3B activated parameters against \(32\times10^9\times 2\ \mathrm{B} = 64\) GB for 32B (derived; this ignores the KV cache and the larger set of experts touched at bigger batch sizes). It is not a controlled RL-versus-distillation experiment, since base model, data, size and architecture all differ. It does show what a distilled MoE can reach against an RL-trained dense model of the previous generation, at about a tenth of the activated weight traffic.
7. Qwen3 beside R1: one skeleton, different choices
Module 27 reads R1 in full. Table 2 lists only what the comparison needs. The skeleton is shared: a small long-CoT cold start, a verifier-driven reasoning RL stage, an SFT step built by rejection sampling from that RL model, a broader final RL stage that admits learned rewards, and distilled small models. The choices inside each box differ, and so does how much each report prints.
| Stage | DeepSeek-R1 | Qwen3 flagship |
|---|---|---|
| Cold start | "Thousands" of long-CoT examples in first-person style; readability is the stated motive (§3, Supp. B.3.2) | QwQ-32B traces after query and response filters; count not reported; kept small on purpose (§4.1) |
| RL prompts | 80K reasoning prompts by Table 4 (26K math, 17K code, 22K STEM, 15K logic); 88K if the text's 8K bug-fixing problems are added (Supp. B.3.1) | 3,995 query-verifier pairs (§4.2) |
| Reasoning reward | Rule-based accuracy and format, plus a language-consistency term (§2.2, §3.2.1) | Rule-based verifiers (§4.2, §4.4) |
| Reasoning-RL settings | Learning rate 3e-6, KL 0.001, clip ratio printed as 10, 16 samples per question, 32 questions per step, temperature 1, 32,768 tokens (§3.2.1) | None printed; "large batch", "high number of rollouts", off-policy training (§4.2) |
| Middle SFT step | Rejection sampling from the stage-1 RL model: about 600K reasoning (correct only) plus about 200K non-reasoning samples (Supp. B.3.3) | Fusion SFT: thinking data by rejection sampling from the Stage 2 model, plus curated non-thinking data; counts not reported (§4.3) |
| Final RL | 1,700 steps at temperature 0.7; preference reward only in the last 400 steps (§3.2.2) | More than 20 tasks, three reward types; step count not reported (§4.4) |
| Small models | SFT on the 800K samples for 2 to 3 epochs, no RL stage (Supp. B.4.3, F) | Off-policy then on-policy logit distillation from Qwen3-32B or Qwen3-235B-A22B (§4.5) |
| Compute printed | 147K GPU-hours in total, $294K at $2 per H800 GPU-hour (Supp. Table 7) | Only Table 21: 17,920 and 1,800 GPU-hours for the RL and on-policy-distillation arms; hardware and teacher cost not stated (§4.7) |
Two things stand out. The RL data differs by a factor of 20 while the scaffolding is the same, which makes the 3,995-pair set a deliberate bet that a few well-chosen verifiable prompts are enough from a cold-started model, a bet the Qwen3 report does not test. And the R1 authors end their distillation appendix by leaving "the exploration of the RL stage" on top of distilled models "to the broader research community"4 (Supp. F). Qwen3's on-policy phase is one answer in kind, and Table 21 compares it with RL from the same start. The Qwen3 report does not present it as an answer to R1, and the comparison is on one model size and one query domain.
8. What the report does not print
A reader who wants to reproduce or budget a Qwen3-style run needs the items in Table 3. Each row was checked against the post-training text of the report (§4.1 to §4.7); the report's hyperparameter statements elsewhere concern pretraining.
| Stage | Stated | Not reported |
|---|---|---|
| Cold start | Filters, generator (QwQ-32B), six response defects, "minimize" samples and steps (§4.1) | Number of candidates \(N\), dataset size, steps, learning rate |
| Reasoning RL | GRPO; 3,995 pairs; four criteria; entropy kept steady or rising; AIME'24 70.1 to 85.1 in 170 steps at 235B (§4.2) | Pool size and pass-rate thresholds; batch size, rollouts per query, off-policy degree, learning rate, KL, clip range, maximum length; how entropy is controlled |
| Fusion | Data construction and the chat template (§4.3, Table 9) | Sample counts, epochs, thinking-to-non-thinking ratio |
| General RL | Five capability groups; three reward types; judge identity; agent rollouts with execution feedback (§4.4) | The 20-plus tasks; task-to-reward mapping; reward scales and mixing; RM size and data; judge prompt; environments, tools, turn limits; steps |
| Distillation | Two phases; teachers named as Qwen3-32B or Qwen3-235B-A22B; "minimize the KL divergence" (§4.5) | Teacher per student, prompts, steps, KL direction and estimator, correctness filtering in the off-policy phase |
| Compute | 17,920 and 1,800 GPU-hours (Table 21); "1/10" twice | GPU type and count; any other stage's cost; whether the 1,800 includes the teacher |
| Evidence | Table 22 (32B, stage by stage); Table 21 (8B); final-model tables | Flagship stage-wise results; seeds and intervals; the pass@64 estimator; ablations of the 3,995-pair choice, the cold-start size and the reward types |
The most consequential omissions for a practitioner are the batch geometry and the entropy method of Stage 2, because the report's one positive claim, "without any manual intervention on hyperparameters", cannot be tested without them, and the task-to-reward mapping of Stage 4, because the mapping decides where hacking risk sits.
9. What the recipe leaves at the serving layer
For an inference engineer the post-training choices surface as decoding settings and conversation rules. Table 4 collects them from the report's evaluation protocol and from the model cards; the cards are the only source for the practitioner-facing recommendations.
| Source | Mode | Temperature | Top-p | Top-k | Min-p | Presence penalty | Output length (tokens) |
|---|---|---|---|---|---|---|---|
| Report evaluation | Thinking | 0.6 | 0.95 | 20 | not stated | 1.5 on two writing benchmarks only | 32,768; 38,912 for AIME |
| Report evaluation | Non-thinking | 0.7 | 0.8 | 20 | not stated | 1.5 | 32,768; 38,912 for AIME |
| Qwen3-8B card | Thinking | 0.6 | 0.95 | 20 | 0 | 0 to 2, optional | 32,768 for most queries; 38,912 for highly complex math and programming benchmarks |
| Qwen3-8B card | Non-thinking | 0.7 | 0.8 | 20 | 0 | 0 to 2, optional | 32,768 for most queries; 38,912 for highly complex math and programming benchmarks |
| Instruct-2507 card | Non-thinking only | 0.7 | 0.8 | 20 | 0 | 0 to 2, optional | 16,384 for most queries |
| Thinking-2507 card | Thinking only | 0.6 | 0.95 | 20 | 0 | 0 to 2, optional | 32,768 for most queries; up to 81,920 for highly complex math and programming benchmarks |
Three further statements from the Qwen3-8B card matter to a runtime. "DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions". In multi-turn conversations "the historical model output should only include the final output part", with the provided chat template handling that stripping. And enable_thinking defaults to true; with it set to false the model emits no think block and the /think and /no_think soft switches "are not valid"11.
Three consequences are the writer's reading, not claims of the sources. One set of weights now serves two output-length profiles, so a scheduler that mixes the modes sees a bimodal length distribution, the situation Module 1's static-batching waste arithmetic describes. Stripping earlier thinking from the history means the prompt for turn \(n+1\) no longer contains the thinking tokens whose KV entries were computed while generating turn \(n\), so prefix-cache reuse across turns stops where the previous assistant message begins. And the budget, which the report implements by halting the thinking at a threshold and appending a stop string1 (§4.3), is a serving-time action: the runtime must inject text mid-decode and continue from the extended prefix. The 2507 releases ship one mode per checkpoint, which turns the first issue from a scheduling problem into a deployment choice between two models (last section).
Where experts disagree
Qwen3 sits inside four live disputes. Each is stated below as its own set of positions, with the Qwen3 evidence placed where it belongs and no more weight than it can bear.
Disagreement 1: RL or distillation for strong small reasoning models?
Distill: a teacher signal beats RL for the student
Qwen3 team1; DeepSeek-R1 authors4; Thinking Machines10
Qwen3 reports on-policy distillation ahead of RL on all six Table 21 benchmarks at about a tenth of the GPU-hours. R1 found that 10K+ RL steps on Qwen2.5-32B only matched QwQ-32B-Preview while distilling R1 did "significantly better". Thinking Machines reproduces the direction with its own accounting.
- Three groups, one direction (R1's comparison is off-policy SFT distillation, not on-policy), and Table 21 starts both arms from the same checkpoint.
- The cost is paid once per teacher and reused across six students.
- The 17,920 and 1,800 GPU-hours come with no hardware and no statement on whether the teacher is counted (Module 21).
- The teacher is itself the product of a four-stage RL pipeline, so distillation depends on RL rather than replacing it.
- The RL arm's recipe is not described; it may not be a tuned run.
RL remains necessary at the frontier and may close the gap with enough steps
DeepSeek-R1 authors, same appendix4; ProRL authors14
R1's authors add that "advancing beyond the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning". ProRL reports that prolonged RL from a distilled 1.5B model, with KL control, reference resets and diverse tasks, finds strategies the base model cannot reach.
- Distillation cannot exceed the teacher; only RL has a path past it.
- ProRL's 2,000-plus steps is a different regime from a short RL arm.
- Needs a long run: ProRL used four nodes of 8 H100 for about 16K GPU-hours14 (§3.2), the same order as Qwen3's RL arm of 17,920 (hardware unstated), so cost alone does not separate the two.
- No source compares a long RL run with on-policy distillation at matched total compute.
For a small student with an existing strong teacher that shares the tokenizer and an SFT start, distillation is the cheaper route in every comparison available, and Qwen3 shows it at product scale (six models). That is a statement about copying a capability, not creating one: the teachers cost four RL-shaped stages that Qwen3 never prices. Whether RL given Qwen3-sized compute and ProRL-style stabilizers would match OPD is untested. The evidence that would settle it is one starting checkpoint, teacher cost counted, matched total GPU-hours, several seeds, and pass@k over more than 30 problems.
Disagreement 2: does RLVR create reasoning capability or sharpen what exists?
RL sharpens; distillation expands
Yue et al.15; DeepSeekMath's Maj@K versus Pass@K observation5; Qwen3's Table 211
RLVR models beat their base at small \(k\) and lose at large \(k\), and the reasoning paths of RLVR models were already inside the base model's samples. Distillation, unlike RLVR, can introduce new patterns from a teacher. Qwen3's Table 21 has the same shape: RL left AIME pass@64 at 90.0 and 83.3, on-policy distillation raised it to 93.3 and 86.7.
- Yue et al. report the pattern across six RLVR algorithms.
- Gives a mechanism: a teacher's logits carry information RL's one-bit reward cannot.
- Table 21's expansion is one question out of 30 on each AIME set, from one run, with no stated estimator (Modules 20 and 21).
- The RL arm's length and recipe are not given, and the report's RL pass@64 is for a model that started from an already distilled checkpoint.
RL can expand, in a second phase or with prolonged training
ProRL14; the two-stage dynamics view16
ProRL reports gains "inaccessible to base models, even under extensive sampling" and ties them to base-model weakness on the task and to training duration. The two-stage view argues both camps are partly right: an exploitation phase that sharpens, then an exploration phase in which rare good tokens gain probability.
- Explains why short runs (Qwen3's RL stage ran 170 steps at 235B) look like sharpening.
- Results are on a 1.5B model; the two-stage paper is a theoretical reconciliation by its own authors' account.
Much of the gain is prior amplification
Spurious Rewards authors17
On Qwen2.5-Math-7B, randomly assigned rewards gave a 21.4-point MATH-500 gain against 29.1 for true rewards; the effect is model-dependent and absent for other families. The paper's mechanism is a clipping bias that amplifies high-prior behavior.
- A control any pass@k claim should survive.
- The paper's gains are specific to Qwen2.5-Math and minimal or negative for Qwen2.5, Olmo 2 and Llama 3; it tests no Qwen3 model.
Qwen3's own contribution is thin and should be filed as a consistent data point, not a test: a one-question margin from a single run cannot distinguish a real expansion from noise, and the RL arm never had the long horizon where ProRL sees expansion. What the report does support is narrower. Starting from the same checkpoint, a teacher-driven signal gave higher pass@1 at a tenth of the stated cost (hardware and teacher cost not stated). I found no spurious-rewards control on a Qwen3 checkpoint in the sources, and it is among the cheapest experiments that would reduce the uncertainty.
Disagreement 3: verifiers, learned reward models, or judges?
Verifiers only for reasoning; keep learned rewards on a short leash
DeepSeek-R1 authors4
R1 abstains from neural reward models on reasoning tasks because they are susceptible to hacking at scale, and confines the preference reward to the last 400 of 1,700 second-stage steps.
- Hacking was observed: reward rose while reasoning fell (Supp. B.5).
- Silent on open-ended tasks where no verifier exists.
Route by task: rules where possible, a reference judge or an RM elsewhere
Qwen3 team1
Qwen3 keeps rules for precision, adds a 72B judge with a reference "avoiding false negatives", and a preference RM for queries with no reference, over more than 20 tasks.
- Covers instruction following, format, agent and open-ended preference in one stage; ThinkFollow rises to 98.9.
- No hacking analysis, no exposure cap, no task-to-reward mapping; the judge is fixed and older than the policy.
Rubric judges, trained in the loop
DeepSeek-V3.218; DeepSeek-V419
V3.2 uses a generative reward model with per-prompt rubrics for general tasks. V4 uses no scalar reward models and applies RL to the generative judge, which is the actor itself.
- Removes the fixed-judge ceiling, and needs no separate judge model.
- Shared weights mean shared failure modes; the report cannot audit them from inside (Module 16).
Qwen3 is the middle position, and the evidence is not there to rank it. Where a checker exists, all three agree on using it. For the rest, each design trades a different risk: R1 limits exposure, Qwen3 relies on a fixed judge that is a generation older than the policies it scores, V4 gives up independence for scale. The measurement that separates them is judge accuracy on labelled pairs tracked through training, which none of the three reports prints.
Disagreement 4: one fused model, or separate thinking and non-thinking models?
Fuse the modes in one set of weights
Qwen3 report, April and May 20251
One model saves the cost and complexity of deploying separate models and gets budget control as a side effect. At 32B, thinking-mode AIME'24 goes from 83.8 (Stage 2, before fusion) to 81.4 (Stage 4, fused), and the fused model's non-thinking mode scores 31.0.
- One deployment; budgets from a stop string without extra training.
- Table 22 shows a 2.4-point AIME'24 and 2.7-point LiveCodeBench drop in thinking mode, accepted "to enhance the model's overall versatility".
Train Instruct and Thinking models separately
Qwen team's July 2025 decision, as quoted by The Register20; the 2507 model cards1213
The team's post, as quoted, says: "After talking with the community and thinking it through, we decided to stop using hybrid thinking mode." It adds: "We believe that providing better-quality performance is more important than the unification at this moment." The Instruct-2507 card says the model "supports only non-thinking mode"; the Thinking-2507 card says it "supports only thinking mode".
- Each checkpoint is tuned for one length profile; the Thinking-2507 card prints AIME25 92.3 against 81.5 for the original and a longer thinking length.
- Two deployments. The comparison is confounded: three months of reasoning work, longer thinking and possibly other changes came with the split, so the card's gap is not the cost of fusion.
Qwen's own reversal is the strongest evidence, and it is a statement of priorities, not a measurement. The team says it keeps researching hybrid thinking20. DeepSeek's V4 trains three reasoning-effort modes per model19, so multi-mode single-model designs are still being built, and Module 23 compares the mechanisms. What would settle it is the same model trained both ways with equal data and compute. The primary Qwen post was not retrievable here (the X page returned HTTP 402 to the fetch tool, and I did not work around it), so the quotes are taken from The Register's labelled secondary account.
What is changing
- QwQ-32B
Two-stage RL from a cold-start checkpoint, verifiers first, then a general reward model plus rule-based verifiers3.
- Qwen3 release post
Four-stage pipeline announced; the post gives no hardware or cost and no reward details beyond the task categories2.
- Technical report, v1
The only version of the report; the arXiv listing shows none later1.
- Instruct-2507 and Thinking-2507, hybrid mode dropped
Separate checkpoints with a 262,144-token native context; the Thinking card lists "Pretraining & Post-training" and does not mention RL or GSPO121320.
- GSPO paper, v2
States that GSPO contributed to "the latest Qwen3 models"7.
Two threads are open. First, no later Qwen report in the source map documents a post-training recipe. The Thinking Machines post's June 2026 update says the Qwen3-32B and Qwen3-8B-Base models it used "have since been retired" from its platform and that its recipes now use Qwen3.5-9B10; I found no Qwen3.5 technical report in the sources and do not describe that model's training. Second, the Qwen3 models have become the substrate for other groups' distillation studies. The MOPD paper trains a Qwen3-30B-A3B student, and swapping its same-origin RL teacher for Qwen3-235B-A22B hurt8 (§4.4.2); Module 21 §5.1 reads that as a reachability condition that Qwen3's off-policy phase may exist to satisfy, which is that module's synthesis and not a claim of the Qwen3 report. The model snapshots will keep changing; the one stable fact is the April 2025 report, which has not been revised since 14 May 2025.
References
- paperQwen Team (An Yang et al.). Qwen3 Technical Report. arXiv:2505.09388 v1, 14 May 2025 (no later version listed on the arXiv page). https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-27. Used for: §4 introduction and Figure 1 (pipeline, 1/10 GPU hours), §4.1 (cold start), §4.2 (3,995 pairs, four criteria, 70.1 to 85.1 in 170 steps), §4.3 and Table 9 (fusion), §4.4 (general RL, three reward types, agent rollouts), §4.5 (strong-to-weak distillation), §4.6 (evaluation protocol, Tables 11 to 20), §4.7 (Tables 21 and 22, thinking-budget discussion).
- officialQwen Team. Qwen3: Think Deeper, Act Faster. Qwen blog, 29 April 2025. https://qwenlm.github.io/blog/qwen3/. Accessed 2026-10-27. Used for: release date; the four-stage pipeline as announced; the absence of hardware, cost and reward details in the post (extracted by the fetch tool).
- officialQwen Team. QwQ-32B: Embracing the Power of Reinforcement Learning. Qwen blog, 6 March 2025. https://qwenlm.github.io/blog/qwq-32b/. Accessed 2026-10-27. Used for: QwQ-32B's two-stage RL (accuracy verifier, code execution server, general reward model plus rule-based verifiers, "a small amount of steps"); QwQ-32B as an RL-trained model. Quotes are fetch-tool extractions.
- paperDeepSeek-AI (Daya Guo et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 v2, 4 January 2026 (the expanded version). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Used for: cold-start motive (§3, Supp. B.3.2), first-stage settings and language-consistency reward (§3.2.1), second-stage schedule (§3.2.2), RL data Table 4, 800K SFT data (Supp. B.3.3, B.4.3), compute Table 7, distillation as SFT only and the RL comparison (Supp. F, Tables 15 and 16), verifier findings (Supp. G.1). Read in the cached copy.
- paperShao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 v3, 27 April 2024. https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Used for: the origin of GRPO (the algorithm Qwen3 names in §4.2); the Maj@K versus Pass@K observation (§5.2.2).
- paperKhatri, D. et al. The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786 v1, 15 October 2025. https://arxiv.org/pdf/2510.13786v1. Accessed 2026-10-26. Used for: the statements that PPO-off-policy-k "has been used previously by Qwen3" (§3.1) and that Qwen3 uses interruptions (App. A.10), both flagged here as second-hand readings.
- paperZheng, C. et al. (Qwen Team). Group Sequence Policy Optimization. arXiv:2507.18071 v2, 28 July 2025. https://arxiv.org/pdf/2507.18071v2. Accessed 2026-10-26. Used for: the statements that GSPO was applied to "the latest Qwen3 models" (abstract, §5.1, conclusion); no model, stage or date is named.
- paperXiaomi (internship work). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv:2606.30406 v1, 29 June 2026. https://arxiv.org/pdf/2606.30406v1. Accessed 2026-10-26. Used for: the "Mix-RL" classification of the Qwen3 report (§1); the same-origin teacher result on a Qwen3-30B-A3B student (§4.4.2).
- paperLambert, N. et al. (Allen Institute for AI). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124 v5, 14 April 2025. https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: the verifiable-reward stage on IFEval-style constraints (§1), the work Qwen3 cites for rule-based instruction-following rewards.
- blogLu, K. and Thinking Machines Lab. On-Policy Distillation. Thinking Machines Lab: Connectionism, 27 October 2025; update note June 2026. https://thinkingmachines.ai/blog/on-policy-distillation/. Accessed 2026-10-26. Used for: the 60% AIME'24 SFT baseline against Table 21's 55.0, the RL cost taken from the Qwen3 report, and the June 2026 note that the Qwen3 models used were retired and recipes now use Qwen3.5-9B.
- officialQwen Team. Qwen/Qwen3-8B, Hugging Face model card. https://huggingface.co/Qwen/Qwen3-8B. Accessed 2026-10-27. Used for:
enable_thinking, soft switches, recommended sampling parameters, no-greedy warning, output-length recommendations, history handling (Best Practices; extracted by the fetch tool). - officialQwen Team. Qwen/Qwen3-235B-A22B-Instruct-2507, Hugging Face model card. https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507. Accessed 2026-10-27. Used for: non-thinking-only statement, context length, sampling settings, training-stage listing without RL or GSPO. Quotes are fetch-tool extractions.
- officialQwen Team. Qwen/Qwen3-235B-A22B-Thinking-2507, Hugging Face model card. https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507. Accessed 2026-10-27. Used for: thinking-only statement, AIME25 92.3 against 81.5 for the original, context length, sampling and output-length settings, training-stage listing without RL or GSPO. Quotes are fetch-tool extractions.
- paperLiu, M. et al. (NVIDIA). ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv:2505.24864 v1, 30 May 2025. https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-26. Used for: the expansion claim (abstract), the 2,000-plus step run (§1), 4 nodes of 8 H100 for about 16K GPU-hours (§3.2).
- paperYue, Y. et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837 v5, 24 November 2025. https://arxiv.org/pdf/2504.13837v5. Accessed 2026-10-26. Used for: the pass@k sharpening result and the distillation contrast (abstract; abstract-level read).
- paperThe Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View. arXiv:2510.04028 v1, 5 October 2025. https://arxiv.org/abs/2510.04028. Accessed 2026-10-26. Used for: the exploitation-then-exploration reconciliation (abstract; abstract-level read).
- paperShao, R. et al. Spurious Rewards: Rethinking Training Signals in RLVR. arXiv:2506.10947 v2, 25 February 2026. https://arxiv.org/pdf/2506.10947v2. Accessed 2026-10-26. Used for: the 21.4 versus 29.1 MATH-500 result on Qwen2.5-Math-7B, the model-family dependence, the clipping-bias mechanism (abstract, §1).
- paperDeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556 v1, 2 December 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: the generative reward model with per-prompt rubrics for general tasks (§3).
- paperDeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348 v1 (cited by title and id; the arXiv API date and the id month disagree, so no date is given). https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: no scalar reward models and the actor as generative judge (§5.1.1); three reasoning-effort modes per model with distinct length penalties and context windows (§5.1.1).
- analysisMann, T. Alibaba admits Qwen3's hybrid-thinking mode was dumb. The Register, 31 July 2025. https://www.theregister.com/2025/07/31/alibaba_qwen3_hybrid_thinking/. Accessed 2026-10-27. Used for: the Qwen team's July 2025 statement on stopping hybrid thinking mode, quoted by the article from a post on X that the fetch tool could not retrieve (HTTP 402); secondary source with the outlet's own vendor-benchmark caveat; quotes are fetch-tool extractions. The headline is the outlet's, not Qwen's.