Reinforcement Learning for LLM Post-Training · Module 14

Training-Inference Mismatch: Why the Rollout Engine and the Trainer Disagree on Probabilities, and the Four Families of Fixes

Two engines run the same weights and report different probabilities for the same token, so the "on-policy" batch is quietly off-policy. This module traces where the disagreement enters (reduction order, precision, the sampler, MoE routing, indexer top-k, tokenization), what it does to a run, and the four fix families the 2025-2026 reports use: correct statistically, fix the numerics, remove the source, or mask the damage. You can use it to instrument a rollout and trainer pair, find the dominant source for a new architecture, and pick the cheapest fix that covers it.

Course 45 min read 22 sources As of 2026-10-27 Prerequisites: Module 1, Module 12
The same weights go through a rollout engine and a trainer; six places where the two computations of the same token probability can diverge, ending in the mismatch ratio rho same weights θ_old Rollout engine (vLLM, SGLang) decode: 1 token per step layers: attention, router, matmul LM head to logits sampler: T, top-p, top-k emits y_t and log π_infer(y_t) Trainer (FSDP, Megatron) whole sequence, one pass layers: other kernels, sharding LM head to logits log-softmax, no mask scores y_t: log π_train(y_t) token ids 3 1 5 2 4 6 ρ_t = π_train(y_t) / π_infer(y_t) ≠ 1 1reduction order 2precision, dtype 3different program 4sampling mask 5discrete choices 6token round trip
Figure 1. One checkpoint, two programs. The rollout engine decodes one token per step at a varying batch size, applies a sampler, and returns the probability it used; the trainer evaluates the whole sequence in one pass with different kernels and sharding. Each numbered point is a place a reported divergence enters: reduction order1, precision23, the decode-versus-sequence program2, the sampling mask4, discrete router and indexer choices45, and re-tokenization5. Sections 2.1 to 2.5 take them in turn; Module 12 (Section 1.2, Figure 2) factors the full ratio into this mismatch and the optimizer drift.

1. The mismatch, defined and measured

1.1 What "same weights" does not give you

A rollout engine and a trainer load one checkpoint. For a sampled token \(a_t\) in state \(s_t\), the engine reports \(\log\pi^{infer}(a_t\mid s_t)\) while sampling; run the trainer on the same token ids and it returns \(\log\pi^{train}(a_t\mid s_t)\). The mismatch ratio and its log are $$\rho_t=\frac{\pi^{train}_{\theta_{old}}(a_t\mid s_t)}{\pi^{infer}_{\theta_{old}}(a_t\mid s_t)},\qquad \delta_t=\log\rho_t .$$ This is the \(\rho_t\) of Module 12 (Section 1.2), where the ratio a loss sees is factored exactly into optimizer drift \(r_t\) and this term. Zhong et al. call the same definition "objective-agnostic": it exists before you choose REINFORCE, PPO or GRPO6 (Eq. 1).

Two consequences need separate fixes. The first is a biased gradient. The batch is drawn from \(\mu=\pi^{infer}\) but the optimizer differentiates \(\pi=\pi^{train}\), so the update is \(\mathbb E_{y\sim\mu}[\nabla\log\pi(y)\,R]\) and not the policy gradient \(\mathbb E_{y\sim\pi}[\nabla\log\pi(y)\,R]\)2 (Eq. 3). One update per rollout does not remove it: \(r_t\) is exactly 1 in that regime and the bias still lives in \(\rho_t\). The second is the deployment gap. The weights that maximize reward under \(\pi\) need not maximize it under \(\mu\), the engine that serves the model; a correction inside the loss addresses the first problem and, by construction, not the second2 (Eq. 4).

Qwen's formulation says when a token-level loss is a valid stand-in at all. The exact sequence-level importance weight is a product over tokens, too noisy to use; the token-level surrogate replaces it with a sum of per-token ratios, the first-order expansion \(\prod_t(1+\epsilon_t)\approx 1+\sum_t\epsilon_t\). That holds only while every \(\epsilon_t\) is small, and each token's ratio splits into the training-inference discrepancy and policy staleness, so the surrogate is valid "only when both" are minimized7 (Abstract, §2.3-2.4). The discrepancy is one of the two terms that decide whether the algorithm approximates the right objective, not a rounding error beneath it.

1.2 How large it is, and how each report measures it

Six reports quote a number, each with a different statistic, so the rows of Table 1 cannot be compared with each other. They agree on one pattern: the average gap is small and the tail is not.

Table 1. Reported size of the rollout-trainer gap for the same weights. The statistic differs by row; values marked "read off" are read from a plot by the writer and are approximate.
ReportModel and settingStatisticReported value
MiniMax-M13hybrid-attention model, before and after an FP32 LM headPearson correlation of token probabilities0.987319 before; 0.997135 after
Ring-1T81T-parameter MoE, GRPO baseline, steps 0 to about 405per-token absolute log gap; batch maximumabout 0.027; max about 12 to 15, up to about 24 (read off Fig. 6)
Zhong et al.6Qwen3-1.7B dense, GRPO, vLLM and FSDP\(|\delta_t|\) per batch, mean and maxmean small (plot axis 0 to 0.07); max near 1.0 (Fig. 1; the text calls it a probability difference)
Qi et al.2R1-Distill-Qwen-1.5B, temperature 1.0, no top-p, 32 samples per questionsequence KL[µ‖π]; slope of log ratio per 1K tokensBF16 7.64, slope −1.01; FP16 0.32, slope −0.07 (Fig. 2)
Miles post9Qwen3 dense and MoE, SGLang and Megatron or FSDPK3 KL, rollout against trainerdense 10-5 to 10-3; MoE 10-3 to 10-1 (stated typical ranges)
Thinking Machines1Qwen 2.5-VL instruct 8B, RLVR on Bigmath, 4,096-token rolloutsKL of log-probs, sampler against trainerabout 0.001 with importance weighting; flat 0 when bitwise identical

The means look harmless (about 0.027 nats per token in Ring-1T); the maxima do not (about 12 to 15 nats at the start of Ring-1T's baseline and about 24 at the end, a per-batch maximum near 1.0 in the dense Qwen3-1.7B run). Zhong et al.'s Table 1 traces one sentence token by token. On Qwen3-8B BF16 weights, along one greedy AIME 2024 response, the engine gave the token "that" log-probability −0.694 and the trainer −0.827 (\(\delta_t=-0.133\)), and at that position the trainer's top-1 token was a different string (":\n\n", −0.577): the argmax flipped. The other positions shown agreed to within 0.016.

Schematic of token probabilities in training mode against inference mode, before and after the FP32 LM head fix, with the reported Pearson correlations 0.987319 and 0.997135 0 0 0.5 0.5 1.0 1.0 inference-mode probability training-mode probability before fix: correlation 0.987319 0 0 0.5 0.5 1.0 1.0 inference-mode probability training-mode probability FP32 LM head: correlation 0.997135
Figure 2. Token probabilities from the training-mode code against the inference-mode code, same weights, one point per token. Schematic after MiniMax-M1's Figure 3; point positions are illustrative, the two correlations are the figure's own labels3. The original left panel shows horizontal bands; the text says the mismatch "prevented reward growth"3 (§3.2).

The number to keep from MiniMax-M1's scatter is 0.987319: a correlation that high still stalled reward growth. A Pearson coefficient on token probabilities is dominated by the many tokens near 0 or 1 and says little about the tail (the writer's reading), so the sections below use tail statistics (max, outside-band fraction, sequence-level slope).

1.3 Length turns a small per-token gap into a large sequence-level one

The sequence-level ratio is the product of token ratios, so its log is the sum \(\sum_t\delta_t\). Averaged over sequences drawn from \(\mu\), \(\mathbb E_\mu[\log\rho(y)]=-\mathrm{KL}(\mu\Vert\pi)\), which is \(-kT\) for a per-token KL of \(k\) nats over \(T\) tokens (derived). Qi et al. plot the sequence log-ratio against length for R1-Distill-Qwen-1.5B, sampled at temperature 1.0 with no top-p so that \(\mu\) is comparable to \(\pi\), and fit a slope of −1.01 per thousand tokens in BF16 against −0.07 in FP16, with sequence-level KL[µ‖π] of 7.64 against 0.322 (§3.5, Fig. 2). Reading the slope as \(1000k\) gives a per-token KL of about \(1.0\times10^{-3}\) in BF16 and \(7\times10^{-5}\) in FP16 (derived), which sits inside the dense range in Table 1. Figure 3 draws the two fits.

Expected sequence-level log importance ratio against response length for BF16 and FP16, from the fitted slopes in Qi et al. Figure 2 0 5 10 15 20 25 0 −5 −10 −15 −20 −25 −20.2 nats at 20K −1.4 at 20K BF16: slope −1.01 per 1K, KL 7.64 FP16: slope −0.07 per 1K, KL 0.32 sequence length (thousand tokens) E[log ρ(y)] = −k·T (nats)
Figure 3. Expected sequence-level log importance ratio against response length: straight lines from the slopes fitted in Qi et al.'s Figure 2 (−1.01 and −0.07 nats per thousand tokens)2. The paper's BF16 scatter is wide around its fitted line, with individual long sequences far below it (the plot's axis runs to −50)2 (p. 7). At 20K tokens the BF16 line is at −20.2 nats and the FP16 line at −1.4 (derived).

Why that matters follows from the variance of the weight. If the sequence log-ratio is Gaussian with mean \(-\sigma^2/2\) and variance \(\sigma^2\), the lognormal that keeps \(\mathbb E_\mu[\rho]=1\), then \(\mathrm{KL}=\sigma^2/2\) and \(\mathbb E_\mu[\rho^2]=e^{\sigma^2}\). The fraction of a batch that is effectively usable under sequence-level weights, \(1/\mathbb E[\rho^2]\), is \(e^{-\sigma^2}=e^{-2\mathrm{KL}}\). With Qi et al.'s average KL values that is \(e^{-15.3}\approx2.3\times10^{-7}\) in BF16 and \(e^{-0.64}\approx0.53\) in FP16 (derived). It is a model, not a measurement: it assumes independent tokens and one lognormal, and real weights are heavier-tailed. It does predict the direction of Qi et al.'s result: sequence-level masked IS "suffers from slow convergence due to the high variance" in BF16, and in FP16 the sequence ratio "becomes much more concentrated and stable"2 (§4.2).

Sequence-level importance weight under a per-token KL

Expected log ratio, nats
Std of log ratio, nats
Usable fraction of a batch

Formulas: \(\mathbb E[\log\rho]=-kT\), \(\sigma=\sqrt{2kT}\), usable fraction \(e^{-2kT}\), under the lognormal assumption stated above. The default (100 units, 16K tokens) is BF16-like; 7 units at 16K tokens is FP16-like. A model, not a measurement.

Li et al. give the trust-region version: classical error bounds scale as \(O(T^2)\), and for \(T=4{,}096\) and a maximum per-token KL of \(10^{-4}\) the bound is 1,677 against a maximum reward of 1, "vacuous"10 (§1). Ling Team add that the gap feeds on itself: under smoothness and bias-alignment assumptions their Theorem 1 has the KL between the two engines growing by a constant factor per update, and their baseline curve shows the engine gap and the gradient norm rising together8 (§2.3.2, App. B, Fig. 6).

2. Where two engines on identical weights diverge

Six mechanisms, numbered as in Figure 1: three numerical, one about what is being sampled, and two discrete decisions that turn small noise into a different computation. Comparisons to the serving kernels you know are the writer's; the facts are cited.

2.1 Reduction order: batch size changes the answer

Floating-point addition is not associative, so any change in the order of a reduction can change a result. Figure 4 shows the smallest case worth remembering: with round-to-nearest-even, \((1+2^{-8})+2^{-8}\) is 1.0 in BF16, because each addend is exactly half a unit in the last place of 1.0 and ties go to even, while \(1+(2^{-8}+2^{-8})\) is 1.0078125 (derived). In FP16 both orders agree. The next representable number above 1 is \(1+2^{-10}\approx1.000977\) in FP16 and \(1+2^{-7}\approx1.007812\) in BF16, the eight-fold resolution gap the FP16 argument rests on2 (Table 1).

Two reduction orders for the same three numbers, 1, 2 to the minus 8 and 2 to the minus 8: in BF16 the two orders give 1.0 and 1.0078125, in FP16 both give 1.0078125 order A: (a + b) + c order B: a + (b + c) a = 1 b = 2⁻⁸ c = 2⁻⁸ BF16: 2⁻⁸ lost BF16 1.0 a = 1 b = 2⁻⁸ c = 2⁻⁸ 2⁻⁷ exact BF16 1.0078125 BF16 (7 mantissa bits): A and B differ by 2⁻⁷ = 0.0078125 FP16 (10 mantissa bits): A and B agree, 1.0078125 ties round to even; each add rounds once a kernel picks A or B by batch size
Figure 4. The same three numbers summed in two orders. BF16 has 7 mantissa bits and FP16 has 102 (Table 1); results computed by the writer with round-to-nearest-even. A kernel that splits a reduction differently at different batch sizes is choosing between orders like A and B.

The engineering question is who chooses the order. Horace He's analysis for Thinking Machines argues that the common explanation, concurrency and atomic adds, is wrong for inference: the forward pass contains no atomic adds and is "run-to-run deterministic", so identical requests to an identical server give identical results. What varies is the batch size, because load changes it, and the kernels are not batch-invariant1 (§ "Batch invariance"). Three operations carry reductions and each breaks invariance its own way. RMSNorm may split one row across cores when the batch is small. A matmul may use Split-K, or different tensor-core instructions, to fill the GPU. Attention's decode-time Split-KV changes how many chunks the KV is cut into: a KV length of 1,000 cut four ways gives four chunks of 250, a fixed chunk size of 256 gives three of 256 and one of 232, so the trees differ unless the split size, not the split count, is fixed1. Their invariant kernels use one matmul configuration for all shapes and lost about 20% against cuBLAS, with a Triton kernel that is "not an optimized" one (no TMA)1.

The same page measures the effect in the setting where you would least expect it. Qwen3-235B-A22B-Instruct-2507 at temperature 0, prompted "Tell me about Richard Feynman" 1,000 times with 1,000 tokens each, produced 80 unique completions; the most common appeared 78 times, all 1,000 agreed for the first 102 tokens, and at token 103 992 continued with "Queens, New York" and 8 with "New York City". With batch-invariant kernels all 1,000 were identical1 (Experiments). That is nondeterminism within one engine; a trainer, with its own tile shapes and batch, should disagree with the engine at least as much (the writer's inference).

Two orders are specific to RL. The rollout is a decode loop with tiny matrices and the trainer processes whole sequences in large batches, so different kernels get selected9. And the rollout often runs tensor-parallel while the trainer shards with FSDP, so the all-reduce order differs; Zhang et al.'s Tree-Based Invariant Kernels fix one hierarchical order across TP sizes and report bitwise identical results between vLLM and FSDP (abstract only read)11.

2.2 Precision: where the rounding lands

Order matters only because every partial sum is rounded. The format decides how coarse the rounding is. Qi et al.'s thesis is that BF16 (8 exponent bits, 7 mantissa bits) lets "different rounding errors" in two implementations accumulate over a sequence, while FP16's 10 mantissa bits give "8 times more precision" and absorb implementation differences. FP16's range is roughly \(6.1\times10^{-5}\) to \(6.6\times10^{4}\) against BF16's \(1.2\times10^{-38}\) to \(3.4\times10^{38}\); they argue RL no longer needs BF16's range because weight and activation ranges were set in pre-training2 (§3.1, §3.4).

One result matters for an inference engineer: inference precision alone barely moves task scores. On R1-Distill-Qwen-1.5B with a 32K budget, AMC23 was 62.35 (BF16), 63.10 (FP16) and 62.42 (FP32)2 (Table 2). An eval that looks identical across dtypes does not show the dtype is safe for rollouts: accuracy is a coarse average, while an importance ratio reads the probability of one sampled token.

MiniMax-M1 localized the gap by layer-by-layer comparison and found it in the LM head: "high-magnitude activations in the LM head at the output layer" were "the primary source of error", and raising the head to FP32 realigned the probabilities, with the correlation in Figure 2 rising from 0.987319 to 0.9971353 (§3.2). The report adds that the problem "did not appear in smaller, dense models with softmax attention", so the hybrid-attention architecture, or its kernels, was a precondition. The report does not say why the head is sensitive; here is an arithmetic reason that fits (derived, not MiniMax's). \(\log\pi(a)=z_a-\log\sum_v e^{z_v}\), so an error in the sampled token's own logit passes into \(\log\pi\) nearly one-to-one when that token dominates. A BF16 number in \([16,32)\) has spacing \(2^4\cdot2^{-7}=0.125\), so rounding the logit alone moves it by up to 0.0625 nats, a 6% probability change; FP16 spacing there is \(2^4\cdot2^{-10}=0.0156\), an error of at most 0.0078. Large logits put the format's step function directly on the quantity the importance ratio reads.

2.3 A different program for the same function

Inference computes one token per step from a KV cache; training computes all positions in parallel. Qi et al. note that "tokens are generated auto-regressively during inference but are processed in parallel during training", and add different parallelization strategies and precision-sensitive operations such as MoE top-k2 (§2.2). Backends compound it: the Miles post picks FlashAttention-3 for both engines because it achieves bitwise equality between prefill and decode, which suggests (the writer's reading) that other backends' prefill and decode kernels need not agree9. Quantized rollouts are the lever you may care about: a follow-up by Yao et al. is said to find the mismatch particularly severe with INT8 and FP8 rollouts, with TIS still stabilizing training (second-hand, unverified). DeepSeek-V4's post-training uses FP4 quantization-aware training for MoE expert weights and the indexer's QK path; its report does not discuss the rollout-trainer gap there12 (§1).

2.4 The sampler is part of the policy

The first three sources are noise on the same function; this one is a different function. A rollout engine samples from \(\mathrm{softmax}(z/\tau)\) after top-\(p\) or top-\(k\) truncation and renormalization, while the trainer's \(\log\pi\) is the plain log-softmax of \(z\) (Module 2 states the unbiasedness condition). Two consequences follow.

The first is what the engine reports. vLLM's --logprobs-mode has four values (raw_logprobs, processed_logprobs, raw_logits, processed_logits); "raw" is the value before logit processors and "processed" is after all of them, "including temperature and top_k/top_p", and the default is raw_logprobs13. TRL's documented vLLM server command passes --logprobs-mode processed_logprobs and --max-logprobs -114. A ratio built from the default raw value while sampling at \(\tau\neq1\) or with truncation compares the trainer with the wrong behavior policy (derived; check which mode your framework requests).

The second is support. With top-\(p\) truncation the kept set has mass \(Z_t\ge p\) and a kept token's behavior probability is \(\pi(a)/Z_t\), so the truncation alone contributes a per-token factor \(\rho_t=Z_t\in[p,1]\) and tokens outside the kept set have behavior probability 0 and never appear in a batch. At \(p=0.95\) the worst case is \(\ln0.95=-0.051\) per token, \(-51\) nats over 1,000 tokens (derived; far above typical). DeepSeek-V3.2's remedy is to preserve the top-p and top-k masks from \(\pi_{old}\)'s sampling and apply them to \(\pi_\theta\) in training, "ensuring both policies share identical action subspaces", and it reports that combining top-p sampling with this Keep Sampling Mask "effectively preserves language consistency"4 (§3.1). Qi et al. sidestep it for measurement by sampling at temperature 1.0 with no top-p "so that µ is directly comparable to π"2 (§3.5).

2.5 Discrete decisions amplify noise

A continuous function turns a \(10^{-3}\) perturbation into a \(10^{-3}\) change; a top-\(k\) selection can turn it into a different branch. DeepSeek-V3.2 states that discrepancies between inference and training frameworks, "compounded by policy updates, can result in inconsistent expert routing during inference and training even for identical inputs", inducing "abrupt shifts in the active parameter subspace"4 (§3.1). Qwen's MoE decomposition writes each engine's routed experts explicitly, so the discrepancy term includes the two engines picking different experts for the same input7 (Eq. 6). Routing drift under optimizer updates is Module 12's Section 4.4; the concern here is that routing flips with no update at all.

The sparse-attention indexer is the same mechanism one layer over. GLM-5 uses DeepSeek Sparse Attention (DSA)5, where an indexer retrieves the top-\(k\) most relevant KV entries with \(k=2048\), and reports that the retrieved top-\(k\) results "are critical for RL stability"5 (§3.2). Section 6.2 gives the failure.

The last discrete source is not numerical at all. If the rollout service returns text and the trainer re-tokenizes it, token boundaries, whitespace and normalization, truncation and special-token placement can differ from what was sampled, corrupting the alignment between actions and advantages. GLM-5 reports that a token-in-token-out (TITO) gateway that records the engine's token ids and feeds them to the learner is "critical for asynchronous RL training"5 (§4.1.2). Module 18 covers multi-turn trajectories; the point here is that the token id sequence must match before the probability can.

Table 2. Sources of rollout-trainer divergence, what the reports say about each, and which fix family in Sections 4 to 7 addresses it.
SourceMechanismEvidence in the reportsFixes
Reduction ordersplit-K, split-KV, tile shape and TP size change summation order80 unique completions in 1,000 at temperature 01; TP-size invariance11C (invariant kernels)
PrecisionBF16 rounding, large logits at the LM headcorrelation 0.987319 to 0.997135 with an FP32 head3; KL 7.64 to 0.32 with FP162B
Program shapedecode loop and KV cache against full-sequence pass; attention backendM1 hybrid attention only3; FlashAttention-3 chosen for prefill-decode equality9B, C
Samplertemperature, top-p, top-k, logprob modeKeep Sampling Mask4; --logprobs-mode13C
Router, indexer top-ka small perturbation flips a discrete choiceKeep Routing4; non-deterministic top-k collapse5C, A, D
Tokenizationtext round trip changes token idsTITO gateway5C

3. What the mismatch does to a run

If the mismatch were only noise it would average out. The reports that isolate it find that it does not, and how it hurts depends on where \(\pi_{old}\) comes from.

3.1 The zero-mismatch reference experiment

The cleanest evidence is from Zhong et al., who removed the mismatch rather than correcting it. VeXact is a rollout engine whose token log-probabilities are bitwise identical to FSDP's: the same Hugging Face model code in both, batch-invariant kernels with fixed tiling and reduction order (RMSNorm, batched matmul, fused MoE), attention with KV splitting disabled, and chunked prefill, CUDA graphs and pipeline parallelism to recover throughput6 (§3.1). The only difference between a VeXact run and a vLLM run is the mismatch, so any gap between the runs is attributable to it.

On-policy REINFORCE on Qwen3-30B-A3B (DAPO data) shows the vLLM run improving and then degrading after step 280, with training and validation reward falling from 0.574 and 0.293 to 0.255 and 0.067, while the VeXact run reaches 0.753 and 0.5346 (§3.2, Fig. 2). GRPO runs on Qwen3-1.7B separate the two ways of obtaining \(\pi_{old}\). With the trainer's recomputation, reward falls from about 0.87 to roughly 0.40 within 650 steps, partly recovers, then reaches near zero after about step 1,665; with the engine's own log-probabilities (bypass) it degrades once, to roughly 0.4, without collapsing. VeXact holds about 0.936 (§4.1, Fig. 3). Both ways of computing the ratio failed.

They give three observations. First, aggregate KL probes miss the early phase: in recomputation mode K1 and K3 on the PPO ratio stay "nearly flat during the first 700 steps, even though the reward is already entering the degradation phase", and rise only in bypass mode6 (Fig. 4). Second, with the zero-centered contribution \(C(r)=-(r-1)\hat A\) (same gradient as the usual surrogate, zero at \(r=1\)), the trainer-referenced contribution under vLLM separates by advantage sign, with "a larger positive shift for negative-advantage tokens", while the two references coincide under VeXact6 (Fig. 5): recomputation adds a sign-dependent skew, not uniform noise. Third, bypass uses the right denominator but evaluates the numerator in the trainer's numerics, so "the optimizer exploits numerical artifacts in the trainer's forward pass"; in the first 10 steps on GSM8K with Qwen3-1.7B, 17.3% of token log-probability changes pointed against the sign of the advantage when measured after the update, and none did under VeXact6 (§4.1). That is the deployment gap of Section 1.1 made visible: the update moved the trainer's landscape the right way and the engine that generates the next batch saw something else.

3.2 Collapse timelines in the reports

Table 3 lists what the reports observed, at the step counts they give. The settings differ too much for the step numbers to be compared; the pattern is that the mismatch grows, or a norm moves, before or with the fall in reward.

Table 3. Reported collapses and degradations attributed to the training-inference mismatch, as stated by each source. "Early signal" is what the source says moved first.
SourceModel, algorithmWhat happenedEarly signal
Zhong et al.6Qwen3-30B-A3B MoE, REINFORCEreward falls after step 280gradient-norm anomalies accompany the fall (§4.1)
Zhong et al.6Qwen3-1.7B dense, GRPO, recomputationdegradation by step 650; near-zero reward after about 1,665none in K1 or K3 until late
Thinking Machines1Qwen 2.5-VL instruct 8B, Bigmath, no correctionreward collapses; loss spike around step 318KL spike at the same time
Miles post9Qwen3-30B-A3B, GRPOreward drops sharply after step 320gradient norm falls from about 0.07 to 0.02, then K3 KL rises
Ring-1T paper8Ring-mini-2.0 (16.8B total, 0.75B active), GRPO without KLreward collapses after 180 to 200 steps; gradient norm explodesprobability gap grows (App. C, Fig. 10-11)
Qi et al.2R1-Distill-Qwen-1.5B, BF16, GRPO and token TISpeak training accuracy 73% (VeRL) and 84% (Oat), then degrades; token TIS peaks at 82% and 88% before collapsing"growing training-inference mismatch" beforehand (§4.2)
GLM-55DSA model, non-deterministic top-k in the indexer"drastic performance degradation ... after only a few steps"sharp entropy drop

Two caveats change how to read the table. The Miles post, from the Miles and SGLang RL teams, reports the opposite experience on its own stack: across more than 300 runs in settings "traditionally notorious for instability" it rarely saw a consistent collapse and never saw one in the post-training of GLM 4.5, 4.6 and 4.7; one MoE trajectory did collapse and became the test case for its mitigations, and on a dense Qwen3-4B-Base run the K3 KL rose after about 600 steps while reward stayed stable9. These are the authors' own unreviewed observations; they bound the claim: the hazard's strength depends on the stack. Most entries are small or mid-size models on math tasks; the one trillion-parameter entry, Ring-1T, shows the baseline's gradient norm and probability gap rising and then the stabilized run, and its own limitations section says IcePop "does not achieve perfect training-inference consistency"8 (§5).

4. Family A: correct the gradient statistically

The first family leaves both engines alone and re-weights each update by the mismatch ratio so the expected gradient is again the trainer policy's. It is the cheapest to adopt, a change to the loss, and the reports disagree about whether it is enough.

4.1 Four forms of the same weight

Write \(\rho_t=\pi^{train}_{old}(a_t)/\pi^{infer}_{old}(a_t)\) as in Section 1.1. The estimator that is exactly unbiased multiplies the whole gradient by the sequence ratio \(\rho(y)=\prod_t\rho_t\)2 (Eq. 5). Its variance is the problem of Section 1.3, so every deployed variant trades some bias for variance, along two axes: what happens to a large weight, and which unit carries it.

  • Truncate or mask. Truncated importance sampling (TIS) caps the weight, \(\min(\rho,C)\), so a large ratio is damped but still contributes2 (Eq. 6, 9). Masked importance sampling (MIS) zeroes samples whose ratio leaves a range and keeps in-range ratios2 (Eq. 7, 10). TIS is credited to Yao et al. and MIS to Liu et al. (second-hand, posts unread). TRL implements both with an optional lower bound, "inspired by" IcePop (Section 7)14.
  • Token, sequence or in between. Per token the weight is cheap and, in the Miles post's words, "mathematically biased"; the sequence product is unbiased but has "extreme variance"; the geometric mean of the token weights sits between them and is less numerically unstable than the product9.

Shipped settings differ. TRL's GRPO trainer enables sequence-level masked IS by default for vLLM generation, with \(C_{max}=3.0\) and no lower bound15. Olmo 3 multiplies the loss by a token-level truncated weight, taking the denominator from the probabilities vLLM returns, with a cap of 2.0 in its 32B Think and RL-Zero runs and none listed for its 7B Think and 7B Instruct runs16 (§4.4.1, Table 49). Qwen's 30B MoE study caps the token weight at 57 (§4.2). Zhong et al. note why token-level TIS works: multiplying the PPO ratio \(r=\pi_\theta/\pi^{train}_{old}\) by \(\rho\) gives \(\pi_\theta/\pi^{infer}_{old}\), the bypass ratio, so TIS turns the recomputation loss into the bypass loss with a cap6 (§4.2). Module 12 (Section 1.2) shows where each design places the factor.

4.2 The price is a forward pass, not a mismatch

Qi et al. observe that the GRPO-style corrections need \(\pi^{train}_{old}\) for the sampled tokens, an extra trainer forward pass before the update, "approximately 25%" more training cost if a backward pass costs twice a forward pass2 (§1, §2.1.1). The arithmetic is Module 1's: a forward pass is about \(2N\) FLOPs per token and forward plus backward \(6N\), so the extra \(2N\) is 33% on top of an uncorrected trainer and 25% of the corrected \(8N\) (derived; I read the paper's 25% as the second). The estimator of Eq. 5 needs no such pass: it has one ratio \(\pi_\theta/\mu_{\theta'}\) and \(\pi_\theta\) comes from the loss's own forward pass. The extra pass is the price of keeping PPO's drift ratio \(r\) separate from \(\rho\) (the writer's reading of Eq. 5-10).

What that costs in wall-clock depends on whether the trainer is the bottleneck, Module 1's question. Olmo 3's footnote gives a 1,000 s step with 125 s of training16 (§4.4.3, footnote 41; Module 1, Section 5.1, reconciles it with the text's 75% idle figure): serialized, the extra 41.7 s is 4.2% of the step, and in an asynchronous design it hides behind generation until the learner's busy fraction rises from 12.5% to 16.7%. Tulu 3's 405B RLVR iteration took about 550 s of inference, 25 s of weight transfer and 1,500 s of training17 (§8.1); there the extra 500 s is 24.1% of the 2,075 s. All of it assumes a forward-only pass runs at the same utilization as the training pass.

Seconds per RL iteration for Olmo 3 and Tulu 3 405B with the extra forward pass for importance-ratio recomputation added as a derived segment 0 500 1,000 1,500 2,000 2,500 seconds per iteration Olmo 3 (asynchronous) 1,000 s step; +41.7 s if serialized Tulu 3 405B (synchronous) 2,075 s iteration; +500 s 875 s 125 s train +41.7 550 s 1,500 s train +500 s 25 s sync generation / inference weight transfer training pass extra forward pass (derived)
Figure 5. Seconds per iteration for two reported runs with the extra forward pass for the mismatch ratio added (red, derived as one third of the training time at equal utilization). Olmo 3: 1,000 s step with 125 s of training, from footnote 4116. Tulu 3 405B: about 550 s, 25 s and 1,500 s17. The runs differ in design (asynchronous against synchronous), so the bars are not a comparison of the two recipes.

Wall-clock cost of the extra forward pass

Step time before
Step time after
Slowdown

Defaults are Olmo 3's 1,000 s step (875 s generation-bound, 125 s training) and one third for the extra pass. The asynchronous branch is \(\max(T_{gen},T_{train})\), a simplification that ignores weight sync and staleness (System Design Modules 37 and 38).

4.3 What the correction does not buy

Qi et al.'s "sanity test" gives the controlled comparison. They keep MATH problems on which R1-Distill-Qwen-1.5B scores 20% to 80% over 40 samples (1,460 questions), so a reliable algorithm should approach 100% training accuracy; each step has 64 questions with 8 rollouts and 4 gradient steps, at \(C=3\) for the IS variants2 (§4, §4.1). In BF16 the plain GRPO baseline peaks at 73% (VeRL) and 84% (Oat) before degrading, and token-level TIS "prolongs training slightly" and collapses after 82% and 88%. GSPO, which does not use the engine's probabilities at all, lasts longer than token TIS but its gradient norm became NaN after 1,200 steps in VeRL. Only sequence-level MIS stays stable in BF16, and it converges slowly and plateaus: 95% training accuracy against 99% in FP16 and 34% against 39% on AIME 2024, which they read as the deployment gap2 (§4.2). The bias-variance reading is theirs: low-variance, high-bias estimators (GRPO, token TIS, GSPO) are fast and collapse, and low-bias, high-variance ones (sequence-level IS or MIS) are stable and slow.

Other groups find the correction necessary and, with care, close to sufficient. Qwen's 30B MoE study reports that on-policy the basic policy gradient with the correction is the most stable, that removing it causes "rapid training collapse and a sharp drop in entropy", and that length normalization, used by GRPO and CISPO, is suboptimal because it breaks the first-order approximation7 (§4.3). Zhong et al. find that a token-level cap followed by sequence-level rejection on \(\rho\), not on drift, "can closely track" the zero-mismatch run, but that it "can only suppress already-generated samples and may also discard useful learning signals", and that without a zero-mismatch reference the thresholds are "largely ad hoc"6 (§4.2). The family fixes the bias at the price of variance and a forward pass; it does not touch the deployment gap.

5. Family B: change the numerics

The second family shrinks the disagreement at its size rather than correcting for it. Reduction order creates a difference; mantissa width and the places where logits are rounded set how large that difference becomes (the writer's synthesis of Sections 2.1 and 2.2). Three moves appear in the reports.

An FP32 LM head. MiniMax-M1 raised the output head to FP32, and the correlation in Figure 2 went from 0.987319 to 0.997135 and "remained stable throughout training, enabling successful reward increase"3 (§3.2). ScaleRL adopted FP32 logits in both generator and trainer after finding that it lifted the fitted asymptote \(A\) of a baseline recipe from 0.52 to 0.61, then found the effect nearly invisible inside the full recipe on a dense 8B model, where run-to-run noise on \(A\) is about 0.02, yet useful for GRPO and DAPO-style losses and for a 17B-by-16-expert MoE, where removing it reduced scalability18 (§3.2, Fig. 5b; leave-one-out and Scout results, Fig. 7 and 8b). Qi et al. say the FP32 head alone "is shown to be insufficient to prevent training collapse", citing Yao et al. and Liu et al., whose posts I could not read; that is second-hand2 (§2.2). The findings are compatible if the head is one of several sources whose weight depends on the architecture, which is how MiniMax states it. The reports give no overhead figure for the head.

FP16 in place of BF16. Qi et al. switch both engines to FP16 and report a sequence-level mismatch "approximately 24× smaller" (KL 7.64 to 0.32, Section 1.3), with no change to the algorithm: under FP16 the five algorithms they compare are "almost indistinguishable", and the plain unbiased sequence-level estimator of Section 4.1 becomes usable2 (§3.5, §4.3). Their ablation with vLLM and FSDP separates the engines: BF16 training with FP32 inference is fully stable but the FP32 inference is "nearly three times slower", while FP16 in both is the most stable and loses no speed. They extend it to a Qwen3-30B-A3B-Base MoE, LoRA, Qwen3-14B-Base with DAPO and OctoThinker-3B, with BF16 LoRA collapsing near step 600, across two frameworks, VeRL and Oat2 (§4.4, §5). Their own limits: FP16 is not claimed "universally optimal", very large models may overflow its range, and FP16 needs loss scaling, which adds a global synchronization to check for overflow in distributed training2 (§3.2, §6). Zhong et al. credit a 2026 TRL-team write-up (Dirhoussi et al.) with the same FP16 finding; I did not locate that write-up6 (§5). The measured 24-fold reduction is larger than the 8-fold finer resolution of FP16 mantissas that one rounding would explain; the paper does not decompose the difference (derived comparison).

Higher precision on one side bought stability at triple the inference cost, a poor trade for the engine that is already the bottleneck in Module 1's cost model.

6. Family C: remove the source

The third family makes the two computations agree exactly. For the continuous sources that means bitwise-identical kernels; for the discrete ones it means reusing the engine's decision in the trainer or making the selection deterministic.

6.1 Bitwise identical engines, and what they cost

Thinking Machines' invariant kernels plus a modified trainer gave bitwise identical results between sampling and training and "0 KL divergence" in an RLVR run on Bigmath with a Qwen 2.5-VL instruct 8B policy and 4,096-token rollouts, where the same run without a correction lost reward and with importance weighting held KL near 0.0011. The price in their test (Qwen-3-8B, one GPU, 1,000 sequences of 90 to 110 tokens) was 26 s for default vLLM, 55 s for unoptimized deterministic vLLM and 42 s with an improved attention kernel: 2.1 and 1.6 times (derived)1. Table 4 collects the follow-ups and the overheads each reports.

Table 4. Reported cost and scope of bitwise-identical rollout and training. The settings differ; the overheads are each source's own and are not comparable across rows.
SourceSettingResultReported cost or limit
Thinking Machines1vLLM with FlexAttention, batch-invariant ops1,000 of 1,000 completions identical; KL 0 in RL26 s default; 42 s and 55 s deterministic (1.6x and 2.1x); matmul about 20% below cuBLAS
SGLang19deterministic mode, FlashInfer and FlashAttention 3, CUDA graphsreproducible RL runs on Qwen3-8B with slime34.35% average slowdown, range 25% to 45%; 61.5% quoted for the earlier kernels
vLLM and TorchTitan20Qwen3 1.7B, GSM8K, synchronous, one hostkl_div exactly 0.0; higher reward in fewer steps2.4x slower RL run; two copies of the model code are "fragile"
Miles Truly On Policy9SGLang with FSDP or Megatron; FlashAttention 3, DeepGEMM, torch.compilebitwise identical log-probs, dense models"some performance loss", unquantified; MoE not covered; no better reward seen on dense
Zhong et al. (VeXact)6rollout engine matched to FSDP on VeRLstable where vLLM collapses (Section 3.1)throughput recovered by chunked prefill, CUDA graphs; no figure quoted
DeepSeek-V412end-to-end batch-invariant and deterministic kernelsbitwise alignment across pre-training, post-training and inference is a design goalauthors' claim: overhead of batch-invariant decoding "negligible"

DeepSeek-V4 is the largest public case. Split-KV is not allowed, since it would leave a partially filled last wave of SMs, so decode attention uses two kernels: one computes a whole sequence in one SM for full waves, and a second spreads one sequence over several SMs for the last partial wave, with its accumulation order designed to match the first and distributed shared memory within a thread-block cluster for the exchange. Matmul replaces cuBLAS with DeepGEMM and drops split-k in most cases, with optimizations that "match or even surpass" standard split-k in most major scenarios (the authors' claim). The backward pass gets per-SM accumulation buffers in attention and token-order pre-processing in MoE12 (§3.3). The report gives the kernel design, not a rollout-mismatch measurement.

Tensor-parallel size is the remaining reduction-order gap: TBIK's tree-structured kernels make results independent of it11. The Miles post is explicit about the burden: its mode requires invasive changes to Megatron and FSDP, is hard to reproduce, and every architecture, parallelism choice and hardware target adds complexity9.

6.2 Discrete choices: replay or determinize

For MoE routing, DeepSeek-V3.2 stores the experts used at sampling and enforces them in training ("Keep Routing", in use since DeepSeek-V3-0324)4 (§3.1); Rollout Routing Replay (R3) is the same idea, recording the inference engine's routing and replaying it in training21. Qwen's analysis finds a cost beyond storage. On-policy, R3 reduced the training-inference KL and still gave no gain, and R3 without the importance correction "still fails rapidly", from which they conclude that replay "can alter the original target policy and introduce bias"7 (§4.3). Replay lowers the measured gap and changes what is being optimized, a reason to keep the correction alongside it. Module 12 (Section 4.4, Figure 6) covers the other branch, GSPO's sequence average.

GLM-5's indexer is the case where replay is not available. The report judges storing the indexer's top-\(k\) indices at every token position "clearly impractical", because \(k=2048\) is "much larger than the k typically used in MoE" and the storage and the traffic between engines would be "enormous"; for scale, at an MoE top-\(k\) of 8 a record would hold 256 times as many indices per decision (illustrative arithmetic, the report gives no figure)5 (§3.2). What worked instead was determinism: replacing the non-deterministic CUDA and TileLang top-\(k\) operators in the training engine's indexer with torch.topk, "slightly slower but deterministic", which "yields substantial RL gains", where the other operators "caused drastic performance degradation during RL after only a few steps, accompanied by a sharp drop in entropy". The indexer's parameters were also frozen during RL to prevent unstable learning in the indexer5. The report does not say which pair of passes disagreed, rollout against trainer or two trainer passes, so what it demonstrates is that a non-deterministic discrete selection is enough to break a run, and that a deterministic one removes the problem at small cost.

Schematic of GLM-5 reasoning RL with two top-k operators in the sparse-attention indexer: with a non-deterministic operator entropy drops sharply and performance degrades after a few steps, with torch.topk training proceeds non-deterministic top-k (CUDA, TileLang) a few steps RL steps (no values in the report) torch.topk (deterministic) RL steps (no values in the report) entropyperformance
Figure 6. Schematic of the GLM-5 indexer finding. Left: with non-deterministic top-k operators entropy drops sharply and performance degrades "after only a few steps". Right: with torch.topk RL proceeds. The report gives the qualitative description only5 (§3.2); no step counts or values are drawn from it, and the curve shapes are illustrative.

7. Family D: mask what the engine's own probabilities flag

The fourth family does not try to make the estimator unbiased; it uses the engine's returned probability to decide what the update should not learn from. It borders family A (IcePop is a two-sided token-level mask on \(\rho\) that passes in-range ratios through); the difference is intent, a trust region on the data and not a correction of the expectation. Module 12 (Sections 5.2, 5.3) writes the IcePop and DeepSeek-V3.2 masks in one form; this section adds the evidence on thresholds.

IcePop comes from Ling Team's Ring-1T report. It masks tokens whose \(\rho\) leaves \([\alpha,\beta]\) and passes the rest through, and for Ring-1T uses \(\alpha=0.5\), \(\beta=5\) with a learning rate of \(2\times10^{-6}\), no KL, temperature 1.0, 480 prompts times 8 rollouts and responses up to 65,536 tokens8 (§2.3.2, §2.3.4). On Ring-mini-2.0 (16.8B total, 0.75B active) it beat TIS on AIME25 (Avg@64), lifting the base score of 63% by "over 14%" while vanilla GRPO fell by step 150; at the defaults about 1 to 2 per thousand tokens were masked, skewed toward high entropy8 (§2.4.1, App. C, Fig. 5, 12). The sensitivity result is the part to carry over: a narrow \([0.5,2.0]\) "immediately destabilizes training", the default \([0.5,5.0]\) stabilizes it, and a wider \([0.4,5.0]\) also stabilizes but admits higher-probability tokens8 (App. C, Fig. 13). GLM-5 ships \(\beta=2\), the range Ring-mini-2.0 found unstable, in a reasoning stage run entirely on-policy with a different model and stack5 (§3.2). The two do not conflict: the band is a property of the stack, to be set against a measured distribution of \(\rho\) (Section 9).

DeepSeek-V3.2 masks whole sequences with negative advantage whose divergence from the engine's returned \(\pi_{old}\) exceeds \(\delta\), reasoning that "highly off-policy negative samples can be detrimental"; \(\delta\) and any ablation are not given4 (§3.1). Because the engine's probability is the reference, the mask sees drift and mismatch together. Li et al. give a rationale for sequence-level masks: their bounds depend on the maximum token-level divergence, "a sequence-level quantity that cannot be controlled by token-independent methods like PPO clipping", and the classical \(O(T^2)\) growth falls to \(O(T^{3/2})\) and \(O(T)\). On Qwen3-8B-Base with vLLM and FSDP they report that PPO clipping at (0.8, 1.28) "exacerbates instability" against token-level IS, while masking with \(\delta=0.05\) (max divergence) or 0.001 (average) keeps the log-perplexity gap bounded10 (§1, §5, Fig. 1-2); authors' claims, one benchmark. Zhong et al. find that rejecting on the mismatch ratio \(\rho\) beats rejecting on the drift ratio, and that a token-level cap before the sequence-level rejection tracks the zero-mismatch run best6 (§4.2). The Miles post lists a veto (discard a sequence if any token's old probability falls below a threshold such as \(10^{-6}\)) and one MoE rescue, where an upper TIS bound of 2.0 collapsed and 1.5 did not, from a single trajectory in more than 300 runs9.

GSPO's sequence-level ratio averages token noise, and its paper argues that sequence likelihood tolerates training-inference precision differences, "intuitively", with no experiment (Module 12, Section 4.4)22 (§5.4). Qi et al. give partial support and a limit: in BF16 GSPO outlasted token-level TIS without using the engine's probabilities, then its gradient norm turned NaN after 1,200 steps in one framework2 (§4.2).

Table 5. The four families side by side. "Bias left" is what remains after the fix as the sources report it; costs are each source's own figures (Sections 4 to 7).
FamilyMechanismBias or gap leftReported costUsed by
A. Statisticalweight by \(\rho\): truncate, mask, token or sequence leveltruncation bias or sequence-level variance; deployment gap untouchedone extra trainer forward pass if \(r\) and \(\rho\) are kept apart, about 25% of the corrected cost2; none for a single-ratio formOlmo 3, TRL default, Qwen study
B. NumericsFP32 head; FP16 in both enginesorder-driven gap remains: KL 0.32, not 02head: none reported; FP16: no speed loss reported; FP32 inference about 3x slowerMiniMax-M1, ScaleRL, Qi et al.
C. Remove the sourcebitwise engines; Keep Routing, R3, Keep Sampling Mask; deterministic top-knone when exact; replay can alter the target policy71.6x to 2.4x or 34.35% for kernels; replay storage; torch.topk slightly slowerV3.2, GLM-5, V4, Miles, vLLM, SGLang
D. Maskdrop tokens or sequences outside a trust region set from \(\rho\) or the engine's \(\pi_{old}\)discards signal; thresholds stack-specific1 to 2 per thousand tokens masked at IcePop defaults8GLM-5, Ring-1T, V3.2, TRM

8. Why MoE and hybrid attention are hit harder

The Miles post puts typical K3 KL at \(10^{-5}\) to \(10^{-3}\) for dense models and \(10^{-3}\) to \(10^{-1}\) for MoE, two orders of magnitude more at the top of the range, and describes the mechanism as logit differences selecting different experts9. Qi et al. (§2.2) and Ling Team (§2.3.2) name routing as the aggravating factor28. It is a discrete amplifier: a perturbation of an ulp can flip a top-\(k\) boundary and change every downstream activation for that token (the writer's reading; Section 2.5), and the same logic covers the sparse-attention indexer (Section 6.2). Hybrid attention is the other precondition in the record: MiniMax-M1's gap appeared with lightning attention and not in smaller dense softmax-attention models3 (§3.2). Length multiplies all of them (Section 1.3), and multi-turn rollouts add the token round trip of Section 2.5 (Module 18). The counterweight is the Miles post's observation that its dense models never collapsed, so a dense softmax-attention stack may need only logging and a default mask, as in TRL915.

9. A procedure for a new architecture

This is the writer's synthesis of the reports' techniques, not a procedure any one report prescribes: log a gap statistic that reaches the tail, vary one factor at a time until the gap moves, and match the fix class to the factor that moved it.

Flowchart for diagnosing training-inference mismatch in a new architecture: log the gap, then run five single-factor tests, each pointing to a fix family, and keep a statistical safety net 1 Log the gap on a real batch mean and max |δ|, K3 KL, outside-band fraction, sequence slope Figure 1 points 1 to 6 are candidates 2 Same engine, two batch sizes: outputs differ? greedy decode of one prompt at batch 1 and batch 64 yes reduction order: invariant kernels (C), else correct (A) no, or both: continue 3 Trainer or engine in FP32 or FP16: gap shrinks? re-run the same token ids at higher precision yes precision: FP32 head or FP16 (B) no, or both: continue 4 Sampler transforms off: gap shrinks? T = 1, no top-p, processed logprobs requested yes sampler: processed logprobs, Keep Sampling Mask (C) no, or both: continue 5 Router or indexer picks differ? compare expert ids or top-k index sets per token yes discrete: replay or deterministic top-k (C) no, or both: continue 6 Token ids differ after the round trip? text out, text in against token ids out yes token-in-token-out gateway (C) no, or both: continue 7 Residual gap remains every report that fixed a source still lists a residual yes keep a mask or cap on ρ and keep logging (A, D)
Figure 7. Diagnosis flow. Step 1 uses the statistics of Table 1 and Section 3; steps 2 and 3 follow Thinking Machines' batch-size test1 and MiniMax-M1's precision and layer-by-layer analysis3; step 4 follows V3.2's Keep Sampling Mask and vLLM's --logprobs-mode413; steps 5 and 6 follow V3.2, GLM-5 and R34521. The order is the writer's.

Step 1 is a few lines. The function below takes the engine's returned log-probabilities of the sampled tokens (requested under the same temperature and truncation the sampler used, Section 2.4) and the trainer's log-probabilities for the same token ids, and returns the statistics of Table 1: mean and max \(|\delta|\), the K3 estimate of KL(engine‖trainer) on engine samples, the fraction outside a band, and the mean sequence log-ratio per thousand tokens, whose sign and size read like Qi et al.'s slope. TRL logs the first two when its correction is on14. Alert on the maximum and the outside-band fraction rather than the means (Section 1.2), and compare changes in them with the gradient norm, which moved before or with the reward in the Miles, Zhong and Ring-1T entries of Table 3.

import torch

def mismatch_report(lp_engine, lp_trainer, lengths, lo=0.5, hi=2.0):
    """lp_*: [B, T] log-probs of the sampled tokens, same token ids, same
    sampling transform on both sides; lengths: [B] valid token counts."""
    T = lp_engine.shape[1]
    valid = torch.arange(T, device=lp_engine.device)[None, :] < lengths[:, None]
    delta = (lp_trainer - lp_engine)[valid]          # delta_t = log rho_t
    rho = delta.exp()
    k3 = rho - 1.0 - delta                           # K3 estimate of KL(engine || trainer)
    seq = ((lp_trainer - lp_engine) * valid).sum(1)  # log rho(y) per sequence
    return {
        "abs_delta_mean": delta.abs().mean().item(),
        "abs_delta_max": delta.abs().max().item(),
        "k3_kl": k3.mean().item(),
        "frac_outside_band": ((rho < lo) | (rho > hi)).float().mean().item(),
        "mean_seq_logratio_per_1k_tokens": (seq / lengths * 1000).mean().item(),
    }

Steps 2 to 6 are single-factor experiments, each run in some form by a report: decoding one prompt at two batch sizes (Thinking Machines' test)1; re-running the trainer in FP32, or moving to FP16 as Qi et al. did, to see whether the gap tracks mantissa width2; turning the sampler's transforms off, as Qi et al. did for measurement; comparing expert ids or indexer index sets between engines; and diffing token ids after a text round trip5. If a first layer where hidden states diverge by more than rounding is visible, as in MiniMax-M1's layer-by-layer trace, the fix may be local, as the FP32 head was there3. Whatever moves the gap, the last box stays: Ling Team's limitations say IcePop leaves "underlying numerical discrepancies" as "a latent source of instability"8 (§5), and Zhong et al. find tuned corrections close to, not equal to, the zero-mismatch run6. Fix the source you can, and keep a threshold on \(\rho\) for what is left.

Where experts disagree

The reports agree that the engines disagree and that "same weights" is not "same policy". They disagree about where to spend the effort. Each position below is backed by runs on its own stack, and no report compares them at matched compute and architecture.

Correct it in the loss (family A and D)

Olmo 3 (token-level TIS, cap 2.0)16; TRL's default sequence-level masked IS15; Qwen's 30B MoE study7; IcePop in Ring-1T and GLM-585.

Engines and trainers will always differ, so make the estimator robust to it with a weight or a mask that works on any architecture.

  • No kernel work; one loss change, and IcePop stabilized a trillion-parameter run.
  • Zhong et al. find tuned corrections track the zero-mismatch run closely6.
  • Leaves the deployment gap: sequence-level MIS plateaued at 95% training accuracy against 99% in FP16 and 34% against 39% on AIME 20242.
  • Thresholds are stack-specific: a narrow band destabilized Ring-mini-2.08; the corrections discard signal and are "ad hoc" without a reference6.

Fix the numerics (family B)

Qi et al. (FP16 in both engines)2; MiniMax-M1 and ScaleRL (FP32 LM head)318.

The mismatch is rounding that accumulates; give the format more mantissa, or the head more precision, and the algorithmic patches become unnecessary.

  • No extra forward pass, no algorithm change; five algorithms become "almost indistinguishable" under FP162.
  • Closes the deployment gap the loss-side fixes cannot.
  • FP16 evidence is mostly one group's; range and loss-scaling limits at large scale, which the authors state2.
  • The FP32-head result splits: ScaleRL gains at the asymptote, while Qi et al. cite posts reporting it insufficient (second-hand)2. FP16 leaves KL 0.32, not 0.

Remove the source (family C)

Thinking Machines, vLLM and TorchTitan, SGLang, Miles120199; Zhong et al.'s VeXact6; DeepSeek-V3.2 and GLM-5 for routing, sampling masks and top-k45; DeepSeek-V4's kernels12.

If the two computations are bitwise identical there is nothing to correct, and the model you train is the model you serve.

  • Exactly on-policy: KL flat at 0, and a ground-truth reference for calibrating any correction6.
  • Reported overheads of 1.6x to 2.4x or 34.35%12019; invasive, covers dense models only in Miles, two copies of model code to keep in step920.
  • Routing replay can alter the target policy7.

Mostly benign on a well-built stack

The Miles and SGLang RL post9.

Across more than 300 runs the authors rarely saw a collapse, none in GLM 4.5 to 4.7 post-training, and treat their mitigations as a fail-safe for rare MoE cases.

  • Guards against over-engineering a dense, mature stack.
  • Own stack, unreviewed; dense collapses are reported with vLLM and FSDP by Qi et al., Zhong et al. and Thinking Machines261.
  • Its one collapse in 300 runs shows the tail is real.
Assessment

The mismatch alone, with everything else fixed, can collapse a run: three groups show it and Zhong et al. isolate it with a zero-mismatch control. How often it bites depends on the stack, so Miles's experience and Zhong's can both be true. The positions are layers, not rivals. On a dense softmax-attention stack, log the tail statistics of Section 9 and keep a default mask; if the tail grows, FP16 or an FP32 head is the cheapest lever, with no extra pass and an effect on the deployment gap. For MoE, share the routing and keep a correction, since replay alone changes the objective. For sparse-attention indexers, make the selection deterministic. Reserve bitwise engines for diagnosis and for scales where 1.6x to 2.4x is acceptable. One matched experiment would settle it: a single MoE, a zero-mismatch reference, each family alone and in pairs, and an evaluation on the serving engine. No report has run it.

What is changing

The vLLM and TorchTitan teams list a unified model definition and compilation support as next steps, to remove the fragility of two model copies20. TRL's main branch enables sequence-level masked IS by default and logs the gap15; read its values as a snapshot. DeepSeek-V4 aims at bitwise alignment between training and inference kernels12. Open: quantized rollouts (the evidence here is second-hand), hybrid and sparse attention beyond the two architectures reported, and whether thresholds can come from a bound such as Li et al.'s10 instead of trial. As of 2026-10-27; the Miles post is undated.

References

  1. blogHe, H. and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab: Connectionism, 10 Sep 2025. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/. Accessed 2026-10-27. Used for: batch invariance and the three reduction operations (RMSNorm, matrix multiplication, attention); the 1,000-completion experiment on Qwen3-235B-A22B-Instruct-2507; the performance table (26 / 55 / 42 s); the true on-policy RL experiment (Experiments, "True on-policy RL"). Full page fetched and the quoted numbers checked against it.
  2. paperQi, P., Liu, Z., Zhou, X., Pang, T., Du, C., Lee, W. S., Lin, M. (Sea AI Lab / National University of Singapore). Defeating the Training-Inference Mismatch via FP16. arXiv:2510.26788v1, 30 Oct 2025. https://arxiv.org/pdf/2510.26788v1. Accessed 2026-10-26. Used for: the biased-gradient and deployment-gap formulation (§2, Eq. 3-4); TIS and MIS forms (Eq. 6-10); the extra-forward-pass cost estimate (§1, §2.1.1); the FP16 and BF16 format table (Table 1); sequence-level mismatch against length (Fig. 2, p. 7); the sanity test and results (§4, Fig. 3-5); MoE, LoRA and 14B runs (§5).
  3. paperMiniMax (Chen, A., Li, A. et al.). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585v1, 16 Jun 2025. https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-26. Used for: the training-versus-inference probability mismatch, the layer-by-layer diagnosis, the FP32 LM head and the correlations in Figure 3 (§3.2, p. 8); the repetition-truncation rule (§3.2).
  4. paperDeepSeek-AI (Liu, A., Mei, A., Lin, B. et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Used for: Off-Policy Sequence Masking (Eq. 8-9), Keep Routing and Keep Sampling Mask (§3.1, pp. 7-8).
  5. paperGLM-5 Team (Zeng, A., Lv, X. et al.; Zhipu AI / Tsinghua). GLM-5: from Vibe Coding to Agentic Engineering. arXiv:2602.15763v2 (24 Feb 2026; v1 17 Feb 2026). https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Used for: GRPO with IcePop, β = 2, no KL, on-policy, group 32 (§3.2, Eq. 1); the DSA indexer finding, deterministic top-k, k = 2048, frozen indexer (§3.2, "DSA RL insights"); the TITO gateway and direct double-sided importance sampling (§3.3, §4.1.2).
  6. paperZhong, T., Ling, N., Pi, Y., Wei, Z., Yu, T., Fox, G., Wu, P., Yu, X. (ByteDance / University of Virginia). Diagnosing Training-Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Reference. arXiv:2605.14220v2, 29 Sep 2026 (v1 14 May 2026). https://arxiv.org/pdf/2605.14220. Accessed 2026-10-27. Used for: the token-level discrepancy δt (Eq. 1) and the argmax-flip example (Table 1); the mean-versus-max statistics (Fig. 1); VeXact and the REINFORCE collapse (§3, Fig. 2); recomputation versus bypass, the KL-estimator finding, the zero-centered loss contribution and the 17.3% misaligned-direction figure (§4.1, Fig. 3-5); the TIS and rejection-sampling ablation (§4.2, Fig. 6).
  7. paperZheng, C., Dang, K., Yu, B., Li, M., Jiang, H., Lin, J., Liu, Y., Lin, H., Wu, C., Hu, F., Yang, A., Zhou, J., Lin, J. (Qwen Team, Alibaba Inc.). Stabilizing Reinforcement Learning with LLMs: Formulation and Practices. arXiv:2512.01374v3, 3 Dec 2025. https://arxiv.org/pdf/2512.01374. Accessed 2026-10-27. Used for: the token-level objective as a first-order approximation (§2.2-2.4, Eq. 1-5); the MoE decomposition (Eq. 6); Vanilla and Rollout Routing Replay, R2 and R3 (§3.2); on-policy and off-policy results (§4.3-4.4). Source added beyond the researcher's map.
  8. paperLing Team (Inclusion AI). Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model (Ring-1T). arXiv:2510.18855v2, 25 Oct 2025. https://arxiv.org/pdf/2510.18855. Accessed 2026-10-27. Used for: IcePop (§2.3.2, Eq. 1-3, Theorem 1), the recipe α = 0.5, β = 5, 480 prompts × 8 rollouts (§2.3.4), Fig. 5-6 (§2.4.1), the sensitivity analysis and clipping statistics (App. C, Fig. 12-13), limitations (§5). Source added beyond the researcher's map.
  9. blogMiles / SGLang RL team (contributors listed in the post's acknowledgments, ByteDance, SGLang RL Team, RadixArk Miles Team). Let Speed Be With Stability: All-In-One Solution to Training-Inference Mismatch with Miles. Awesome-ML-SYS-Tutorial repository, rlhf/slime/mismatch/blog-en.md; no date shown. https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/slime/mismatch/blog-en.md. Accessed 2026-10-27. Used for: the list of mismatch sources, K3 KL magnitudes for dense and MoE models, the Truly On Policy mode and its stated limits, the three-policy (decoupled PPO) picture, aggregation levels, and the four-configuration MoE rescue experiment. Raw markdown fetched and the numbers checked; the experiments are the authors' own and unreviewed. Source added beyond the researcher's map.
  10. paperLi, Y., Liu, J., Xu, J., Tong, Y., Li, Z., Liu, Q., Wang, B. (Fudan University / CUHK-Shenzhen). Trust Region Masking for Long-Horizon LLM Reinforcement Learning. arXiv:2512.23075v5, 26 Jun 2026 (v1 28 Dec 2025). https://arxiv.org/pdf/2512.23075. Accessed 2026-10-27. Used for: the three sources of divergence and the O(T2) bound arithmetic (§1); sequence-level masking (§4); the log-absolute-perplexity gap (§5, Eq. 25) and Fig. 1-2. Source added beyond the researcher's map; it is the arXiv version of the group whose Notion post (not readable) Qi et al. call Liu et al.
  11. paperZhang, Z., Ding, X., Yuan, J., Liu, R., Mao, H., Xing, J., Liu, Z. Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch (TBIK). arXiv:2511.17826v2, 29 May 2026 (v1 21 Nov 2025). https://arxiv.org/abs/2511.17826. Accessed 2026-10-27. Used for: abstract only, read through the arXiv API: tensor-parallel size as a source of mismatch, Tree-Based Invariant Kernels, the vLLM-to-FSDP bitwise result. Source added beyond the researcher's map; the full text was not read.
  12. paperDeepSeek-AI (Xu, A., Lin, B., Xue, B. et al.). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348v1. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Used for: batch-invariant and deterministic kernel libraries, the dual-kernel attention strategy, DeepGEMM in place of cuBLAS, abandoning split-k (§3.3, pp. 18-19); FP4 quantization-aware training in post-training (§1). The arXiv id and submission date disagree in the source map (note on date ambiguities).
  13. officialvLLM project. Engine Arguments (stable documentation). https://docs.vllm.ai/en/stable/configuration/engine_args/. Accessed 2026-10-27. Used for: --logprobs-mode, its four values, the default raw_logprobs, and the definition of raw versus processed. Source added beyond the researcher's map.
  14. officialHugging Face. TRL documentation: GRPO Trainer (docs/source/grpo_trainer.md, main branch; no version stated). https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Used for: the logged metrics sampling/sampling_logp_difference and sampling/importance_sampling_ratio; the default sequence_mask mode; truncation and masking variants; the vLLM server flag --logprobs-mode processed_logprobs; the pointer list to the mismatch literature ("Dealing with the Training-Inference Mismatch"). Main-branch documentation, a snapshot.
  15. repoHugging Face. TRL source: trl/trainer/grpo_config.py (main branch, raw file; no version stated). https://github.com/huggingface/trl/blob/main/trl/trainer/grpo_config.py. Accessed 2026-10-27. Used for: defaults vllm_importance_sampling_correction=True, vllm_importance_sampling_mode="sequence_mask", vllm_importance_sampling_clip_max=3.0, clip_min=None; the four modes; off_policy_mask_threshold (None by default; the docs' example value 0.5). Main-branch source, a snapshot. Source added beyond the researcher's map.
  16. paperOlmo Team (Ettinger, A., Bertsch, A. et al.; Allen Institute for AI). Olmo 3. arXiv:2512.13961v2 (14 Apr 2026; v1 15 Dec 2025). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Used for: the OlmoRL objective with vLLM probabilities and the truncated importance-sampling cap (§4.4.1, Eq. 1); Table 49 (TIS cap 2.0 for 32B Think and RL-Zero, max asynchrony); the 1,000 s step with 125 s of training (§4.4.3, footnote 41).
  17. paperLambert, N., Morrison, J., Pyatkin, V., Huang, S. et al. (Allen Institute for AI / University of Washington). Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124v5. https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Used for: the 405B RLVR run, 16-way tensor-parallel vLLM inference, about 550 s inference, 25 s weight transfer and 1,500 s training per iteration (§8.1).
  18. paperKhatri, D., Madaan, L., Tiwari, R., Bansal, R. et al. (Meta / UT Austin / UCL / UC Berkeley / Harvard / Periodic Labs). The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786v1, 15 Oct 2025. https://arxiv.org/pdf/2510.13786v1. Accessed 2026-10-26. Used for: the FP32 LM-head ablation, asymptote 0.52 to 0.61 (§3.2, Fig. 5b); the leave-one-out and Scout MoE results (§4, Fig. 7, Fig. 8b).
  19. blogThe SGLang Team. Towards Deterministic Inference in SGLang and Reproducible RL Training. LMSYS Org blog, 22 Sep 2025 (updated 24 Sep). https://www.lmsys.org/blog/2025-09-22-sglang-deterministic/. Accessed 2026-10-27. Used for: the 34.35% average slowdown of deterministic mode (25% to 45% range) against the 61.5% quoted for Thinking Machines' kernels; per-request seeds for non-greedy deterministic sampling. Source added beyond the researcher's map.
  20. blogWasti, B., Ye, W., Rao, T., Goin, M., Zhang, P., Liu, T., Gimelshein, N., Kwon, W., You, K., Li, Z. (vLLM and TorchTitan teams). No More Train-Inference Mismatch: Bitwise Consistent On-Policy Reinforcement Learning with vLLM and TorchTitan. vLLM blog, 10 Nov 2025. https://vllm.ai/blog/2025-11-10-bitwise-consistent-train-inference. Accessed 2026-10-27. Used for: the bitwise-consistent Qwen3 1.7B run on GSM8K, the 2.4x slowdown, KL equal to 0.0, the two-copies-of-model-code caveat, the synchronous single-host setup. Source added beyond the researcher's map.
  21. paperMa, W., Zhang, H., Zhao, L., Song, Y., Wang, Y., Sui, Z., Luo, F. Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers (R3). arXiv:2510.11370v2, 21 Oct 2025 (v1 13 Oct 2025). https://arxiv.org/abs/2510.11370. Accessed 2026-10-27. Used for: abstract only: Rollout Routing Replay records the inference engine's routing and replays it in training. Source added beyond the researcher's map; the full text was not read.
  22. paperZheng, C., Liu, S., Li, M., Chen, X.-H., Yu, B. et al. (Qwen Team, Alibaba Inc.). Group Sequence Policy Optimization. arXiv:2507.18071v2, 28 Jul 2025. https://arxiv.org/pdf/2507.18071v2. Accessed 2026-10-26. Used for: the claim that sequence-level likelihood tolerates training-versus-inference precision differences (§5.4); the Routing Replay baseline (§5.3).