System Design · Module 36
Post-Training Pipelines: SFT/Preference Data Flows, Evaluation Gates, and Model Registries
Post-training is not one training loop with different data — it is several distinct models (policy, reward, sometimes reference and critic) wired into a pipeline, run for several rounds, where an inference-serving technique you already know from the GPU-serving side shows up inside the training job itself. This module walks Meta's disclosed Llama 3 recipe end to end, generalizes it to the RLHF dataflow that frameworks like OpenRLHF implement, and states plainly what is not documented well enough to cover.
1. Post-training is a pipeline of models, not one training loop
Pre-training, as Modules 32–35 cover it, is architecturally simple even at 16K-GPU scale: one model, one loss, one dataset streaming through a 4D-parallel training loop for months. Post-training breaks that simplicity on purpose. Meta's own description of Llama 3 alignment names it directly: the backbone of our post-training strategy is a reward model and a language model
1 — already two models with different jobs, different training signals, and (as later sections show) different hardware access patterns. The general RLHF formulation used by frameworks like OpenRLHF and verl/HybridFlow adds two more: a reference policy (a frozen copy of the pre-DPO/pre-RL model, used to bound how far the policy can drift) and, in PPO-style setups, a critic that estimates value2.
That plurality of models is the reason this module exists as a separate unit from Part D's pre-training modules rather than as a coda to them. A pre-training job's failure modes are about keeping one enormous matrix multiplication healthy across thousands of GPUs (Module 35's subject). A post-training pipeline's failure modes are about correctly routing data and weights between several separately-sized models that come in and out of use at different stages, sometimes on the same GPUs and sometimes on different ones. The systems question is no longer just how do I parallelize one model,
it is how do I schedule several models that depend on each other's outputs, and how much do I let their hardware footprints overlap.
Two consequences follow immediately, and both recur through the rest of this module. First, whatever inference-serving techniques you already know — continuous batching, paged KV-cache allocation, tensor-parallel serving — become directly relevant the moment one of those models is doing autoregressive generation rather than a forward/backward pass, because generation-for-training and generation-for-serving are the same workload on the GPU. Section 3 is the concrete instance: Meta imported PagedAttention, an inference-serving mechanism, straight into Llama 3's training pipeline. Second, because several models are involved, the natural way to describe the pipeline is not a linear script but a dataflow graph — nodes are model computations, edges are data dependencies, and some of those edges require reshuffling data or weights across a differently-partitioned set of GPUs. Section 4 develops that graph view directly from the HybridFlow/verl paper's own formalism, and Section 5 shows it instantiated on real Ray-managed GPUs in OpenRLHF.
2. The Llama 3 recipe: six rounds of SFT, reward modeling, rejection sampling, and DPO
Llama 3 405B's post-training procedure is the single most detailed account any frontier lab has published of what actually happens after pre-training ends1. Figure 1 is the loop; this section is what happens inside each box, in the order Meta's paper presents it, using its own numbers.
2.1 Reward modeling: one model, one job, trained on all the data you have
The reward model (RM) is trained on top of the pre-trained checkpoint using human-annotated preference data1. The annotation process deploys multiple models simultaneously and samples two responses from two different models for the same prompt, so annotators are comparing across model variants (different data mixes, different alignment recipes) rather than just across samples of one model1. Annotators rate preference strength on a four-level scale — significantly better, better, slightly better, marginally better — and, critically, are then asked to edit the chosen response to improve it further. A meaningful fraction of the preference data therefore ranks three responses per prompt: edited > chosen > rejected, not just two1.
Table 6 of the paper breaks the resulting preference corpus down by capability: general English is 81.99% of comparisons, coding 6.93%, multilingual 5.19%, reasoning and tools 5.89%, averaging 3.8 turns and 284.0 tokens of response per example1. Two design choices in how this data is used are easy to miss and matter for what follows. First, RM training uses all preference data collected up to the current round, while DPO training (Section 2.3) uses only the latest batches1 — the RM's job is calibration across the whole history, the policy's job is to move relative to where it is now. Second, the RM's training objective drops the margin term that Llama 2 used, because Meta observed diminishing returns from it once preference data scaled up1 — a small, explicitly-stated instance of a training recipe changing as a direct function of dataset size rather than a fixed hyperparameter choice.
The paper also documents a between-round process that functions as a quality gate on the annotation pipeline itself, not just on the data: we implement a quality analysis and human evaluation process to rigorously assess the data collected, allowing us to refine our prompts and provide systematic, actionable feedback to annotators. For example, as Llama 3 improves after each round, we increase prompt complexity accordingly to target areas where the model lags
1. This is the closest thing the paper documents to an "evaluation gate between rounds" — not a benchmark threshold a checkpoint must clear to advance, but a deliberate escalation of task difficulty timed to the model's measured progress, plus direct annotator feedback loops.
2.2 Rejection sampling and SFT: the policy learns from its own filtered best-of-K
Rejection sampling (RS) is the step that turns the reward model into training data. For each prompt collected during human annotation, Meta samples K — typically between 10 and 30
1 — outputs from the latest chat-model policy, usually the best-performing checkpoint from the previous post-training round or the best checkpoint for a specific capability, and uses the reward model to select the single best candidate1. In later rounds, system prompts are introduced during RS to steer responses toward a desired tone, style, or format, which differs by capability (code responses get instructions about comments and variable naming; general English responses do not)1.
The selected completions, plus other data sources including synthetic data targeting specific capabilities, become the SFT dataset. SFT itself is a standard cross-entropy loss on target tokens with the loss masked on prompt tokens — the largest models are fine-tuned at a learning rate of \(10^{-5}\) over 8,500 to 9,000 steps, a setting the paper reports works well across rounds and data mixes1. Table 7 gives the resulting SFT corpus composition: general English 52.66%, reasoning and tools 21.19%, code 14.89%, exam-like 8.14%, multilingual 3.01%, and long context just 0.11% of examples but with by far the longest average context (38,135.6 tokens) and response (740.5 tokens)1. Across the whole SFT mix the average is 4.7 turns, 535.7 tokens of context, and 310.4 tokens of final response1 — those two context/response numbers reappear in Section 3's derivation.
A second, model-based quality gate runs on this data before it reaches training: a reward-model-based score (top quartile of RM scores counted as high quality) combined with a Llama-3-as-judge score (a three-point scale for general English on accuracy, instruction-following, and tone; a two-point scale for code on bug identification and intent) — samples are kept if either signal marks them high quality, because the paper found the two signals disagree often enough that the union gives the best recall against an internal test set1. A separate difficulty score (via Instag-style intention tagging and a Llama-based difficulty rating) lets Meta prioritize harder examples, and a semantic-deduplication pass — clustering complete dialogues with RoBERTa, then greedily keeping only examples below a cosine-similarity threshold to anything already kept in that cluster, sorted by quality×difficulty — removes near-duplicate examples1.
2.3 Direct preference optimization, and why Meta chose it over PPO
The SFT model is then further aligned with Direct Preference Optimization (DPO) using the most recent batches of preference data from the best-performing checkpoints of the previous round1. Using only the recent batches, rather than the full accumulated history RM training uses, is deliberate: our training data conforms better to the distribution of the policy model that is being optimized in each round
1 — i.e., DPO wants on-policy preference data, RM wants breadth.
Meta states the algorithmic choice explicitly rather than leaving it implicit: we also explored on-policy algorithms such as PPO, but found that DPO required less compute for large-scale models and performed better, especially on instruction-following benchmarks like IFEval
1. Llama 3 uses a learning rate of \(10^{-5}\) and \(\beta = 0.1\), plus two modifications to the standard DPO loss that the paper calls out as necessary at this scale: masking the special formatting tokens (dialog headers and turn-termination tokens) out of both the chosen and rejected sequences in the loss, because leaving them in creates a conflicting objective — the same token appears in both chosen and rejected responses, so the loss simultaneously tries to raise and lower its likelihood, producing tail repetition or premature termination — and adding a negative log-likelihood regularization term on the chosen sequences, scaled by 0.2, to stop DPO training from silently decreasing the chosen response's log-probability while still preferring it over the rejected one1. Finally, Meta averages models obtained from different data or hyperparameter variants at each of the RM, SFT, and DPO stages1 — a cheap way of reducing variance across parallel experiments without additional training.
2.4 What "six rounds" buys
All of the above — reward modeling, rejection sampling, SFT, DPO — is applied six times1. Each cycle collects new preference annotations and new SFT data, sampling synthetic data from the latest models available at that point1. Mechanically, six rounds means six separate reward models (each trained on a strictly larger preference corpus than the last, since RM training accumulates), six rejection-sampling passes over the growing prompt set (each drawing its K completions from a better policy than the round before), and six DPO updates (each trained on that round's freshest preference batches). The loop in Figure 1 is not a fixed-point iteration in the mathematical sense — there is no convergence criterion stated in the paper — it is a fixed schedule, and the escalating prompt complexity described in Section 2.1 is the mechanism that keeps six rounds from training on a static task.
What advances a checkpoint from "the best model of round \(n\)" to the seed for round \(n+1\) is, per the paper's own language, "the best performing checkpoint" — selected by whatever combination of benchmark and human evaluation Meta's evaluation suite runs, which the post-training section does not itemize as a named process, tool, or format. Section 7 returns to exactly what that gap means for anyone trying to reproduce this pipeline's operational tooling, not just its algorithm.
2.5 Capability experts: branching a specialist off pre-training
Figure 1's "SFT" and "DPO" boxes are drawn as if one model runs the whole loop, and for the general-English majority of the data that is accurate. For specific capabilities — code, multilingual, long context, tool use, math and reasoning — the paper describes branching a dedicated expert model off the main pre-training run and continuing to pre-train it on a domain-heavy data mix before ever entering the SFT/DPO loop, then using that expert two ways: to collect higher-quality human annotations in its domain, and as the model rejection sampling draws its K completions from when the prompt is in that domain1. The code expert, for instance, continues pre-training on a mix that is more than 85% domain-specific data, adds long-context fine-tuning to extend it to 16K tokens on repository-level code, and only then goes through the same SFT/DPO recipe as Section 2 describes, with its data mix targeted at code1. The multilingual expert follows the same pattern, branching off pre-training onto a data mix that is 90% multilingual tokens1. Figure 1's loop, in other words, runs multiple times in parallel on different specialist checkpoints, not once on a single generalist model, and rejection sampling routes each prompt to whichever checkpoint (generalist or the relevant expert) is currently best at that capability.
The code expert is also the source of the clearest disclosed number on how much of the SFT data in a modern post-training pipeline is synthetic rather than human-annotated. Because training Llama 3 405B on its own unfiltered generations degraded quality, Meta added execution feedback as a correctness signal: generate a problem description, prompt the model for a solution, then check that solution with a parser/linter (static analysis) and with generated, containerized unit tests (dynamic analysis); when a solution fails, feed the parser/linter/test output back to the model and ask it to revise, repeating until the checks pass or the attempt is discarded1. Meta reports that roughly 20% of solutions were initially incorrect but self-corrected through this loop, and that this single execution-feedback pipeline alone produced on the order of one million synthetic coding dialogues, out of a total of over 2.7 million synthetic examples used in SFT for code1. That number is worth holding onto: for at least this one capability, the volume of model-generated, execution-verified training data dwarfs the volume of anything a human annotator produced directly, which is a data-pipeline fact as much as a modeling one — it is the post-training-stage analogue of the quality/dedup filtering Module 32 covers for pre-training corpora, applied instead to a corpus the model itself is generating.
3. Where PagedAttention leaks into the pipeline
You already know PagedAttention as the mechanism that turned KV-cache allocation from a contiguous-buffer problem into a virtual-memory problem: fixed-size blocks ("pages"), a page table per sequence, and the ability to share physical pages between sequences via copy-on-write3. What matters for this module is not re-deriving that mechanism — it is what changes when the thing calling it is an offline batch-generation step inside a training pipeline instead of an online multi-tenant inference server.
Meta states the adoption directly: to increase the efficiency of rejection sampling, we adopt PagedAttention [Kwon et al., 2023]… together, this leads to a throughput improvement of over 2× during rejection sampling
1. That much is the headline. The mechanism-level detail is where the leak becomes visible, and it runs in two directions at once: Llama 3 gets one of PagedAttention's benefits for free (page sharing across a prompt's K outputs) and explicitly declines another (its dynamic, memory-pressure-driven scheduling), because rejection sampling is not the workload PagedAttention was designed to serve.
The admission-control choice is the more interesting half of the story, because it is Meta explicitly declining one of PagedAttention's headline features. The paper is specific about why: PagedAttention enhances memory efficiency through dynamic key-value cache allocation. It supports arbitrary output lengths by dynamically scheduling requests based on the current cache capacity. Unfortunately, this carries the risk of swap-out when running out of memory. To eliminate such swap overhead, we define a maximum output length and perform a request only if sufficient memory is available to fit an output with that length
1. An online server cannot know in advance how long a generation will run, so it must tolerate the possibility of swap or preemption to keep serving new requests under an SLO. Rejection sampling is an offline batch job with no per-request latency target and a data-dependent but boundable output length, so trading away the elasticity that swap buys you, in exchange for a predictable admission decision that never touches host memory, is a straightforward win. This is the reverse of the usual direction of technology transfer implied by "inference technique leaks into training" — Meta did not just reuse PagedAttention, it re-specified the admission policy around it for a different workload shape.
The other half — PagedAttention also enables us to share the key-value cache pages for a prompt across all corresponding outputs
1 — is the more familiar transfer, and it is a better fit for rejection sampling than for typical multi-tenant serving. In general online serving, most concurrent requests come from different users with unrelated prompts, so there is little prompt-level KV to share unless the platform layers on an explicit prefix-cache (Module 28's subject). Rejection sampling manufactures the ideal case for prompt-page sharing by construction: every one of the K=10–30 samples for a given round starts from the identical prompt, so the entire prompt-length KV cache can be written once and referenced by all K generations, with only the diverging, per-sample continuation tokens needing their own pages.
| Dimension | Generic inference serving | Llama 3 rejection sampling |
|---|---|---|
| Scheduling | Dynamic, based on current cache capacity | Static admission decision at request time |
| Output length | Arbitrary, unknown in advance | Capped at a fixed maximum, checked before admission |
| Memory pressure response | Swap-out to host memory | Swap disabled by construction — never admitted if it can't fit |
| KV page sharing | Only with explicit cross-request prefix caching | Automatic: K samples share one prompt's pages by default |
| Objective | Tail-latency-bounded throughput under an SLO | Maximum aggregate throughput, no latency target |
How much of the reported >2× throughput improvement is plausibly attributable to page-sharing alone, as opposed to PagedAttention's other standard benefit (eliminating the internal fragmentation that a contiguous, pre-reserved KV buffer forces on every request regardless of workload)? Meta does not decompose the number, but its own disclosed data statistics make a bounding estimate possible.
Derivation: memory saved by prompt-page sharing across K rejection samples
Model the KV-cache footprint of one prompt's rejection-sampling batch in units of tokens (pages are a fixed number of tokens each, so token-count is a fair proxy for page-count). Let \(P\) be the prompt/context length shared by all K samples and \(O\) be the length of one sampled completion. Without page sharing — K independent, non-paged KV caches, one per sample — the batch needs:
$$M_{\text{naive}} = K \cdot (P + O)$$With PagedAttention's prompt-page sharing, the \(P\)-token prefix is stored once and referenced by all K samples, so only the K diverging completions need their own pages:
$$M_{\text{shared}} = P + K \cdot O$$The fractional memory saved is:
$$1 - \frac{M_{\text{shared}}}{M_{\text{naive}}} = 1 - \frac{P + K \cdot O}{K \cdot (P + O)}$$Llama 3's SFT-data statistics (Table 7 of the paper) report an average context length of 535.7 tokens and an average final-response length of 310.4 tokens across the full SFT mix1 — the SFT data is exactly the output of rejection sampling, so these are reasonable stand-ins for \(P\) and \(O\). Taking \(K = 20\), the midpoint of the paper's stated 10–30 range1:
$$M_{\text{naive}} = 20 \times (535.7 + 310.4) = 16{,}922 \text{ token-slots}$$ $$M_{\text{shared}} = 535.7 + 20 \times 310.4 = 6{,}743.7 \text{ token-slots}$$ $$1 - \frac{6{,}743.7}{16{,}922} \approx 60\%$$A roughly 60% reduction in per-prompt KV footprint, at a fixed GPU memory budget, translates into a proportionally larger schedulable batch — and in a decode-heavy, memory-capacity-bound regime, throughput scales close to linearly with the number of sequences you can hold concurrently, which is the first-order reason a large memory saving shows up as a large throughput gain. This estimate lines up well with an independent data point: OpenRLHF's own citation of PagedAttention's benefit for the structurally identical "parallel sampling" case (multiple outputs per prompt) reports memory-usage reduction of up to 55%
4. Page-sharing alone is therefore plausibly responsible for most of the reported >2× rejection-sampling speedup, with fragmentation elimination (the benefit PagedAttention provides on any workload, shared prompts or not) contributing the remainder.
4. The general RLHF dataflow: a DAG of models, not a DAG of layers
Llama 3's pipeline (Sections 2–3) has no critic and no PPO rollout loop — DPO's whole appeal, per Meta's own statement, is that it does not need either. Most open-source RL post-training frameworks are built for the more general case that Llama 3 deliberately avoided: PPO-style RLHF, with an actor, a critic, a frozen reference policy, and a reward model all active in the same training step. The HybridFlow/verl paper's formalism for that general case is the right level of abstraction for describing what any RLHF framework — OpenRLHF included — actually has to schedule2.
The formalism: traditional RL can be modeled as a dataflow … a directed acyclic graph (DAG): each node in the RL dataflow represents computation of a neural network … each edge denotes data dependency between NN computations … RLHF dataflow is more complex, with more complicated models involved (e.g., LLMs for the actor/critic/reference/reward models), each running distinct computation, and more diverse data dependencies among them … the edge represents data resharding, which is often a many-to-many multicast
2. Two words in that quote carry the whole systems argument: "resharding" and "multicast." Each of the four models is itself a distributed program (its own tensor/pipeline/data-parallel layout, chosen independently because each model has a different size and workload), so an edge in the DAG is not a simple tensor copy — it is a full re-partitioning of data (or, at the actor-generation boundary specifically, of weights) from one parallelism layout to a different one, fanned out to however many ranks the destination model is sharded across2.
This is exactly why HybridFlow's own paper frames the systems problem as a tension between two paradigms it is trying to reconcile: a single-controller paradigm, where one process holds the whole DAG and dispatches to every model, gives you flexibility to change any one node's data dependencies without touching the others, but pays a large dispatch-overhead tax when the models being dispatched to are themselves billion-parameter distributed programs; a multi-controller paradigm, where each model's own distributed process group drives its own execution, is efficient for the intra-node computation but makes the dataflow rigid — changing how one node's output is consumed requires changing every node downstream of it2. That is not an abstract framework-design concern; it is the direct consequence of Figure 3's structure. A four-node DAG with many-to-many resharding edges has to be coordinated by something, and whatever coordinates it inherits either the flexibility problem or the dispatch-overhead problem depending on which paradigm it picks.
| Dimension | DPO-style (Llama 3) | PPO-style (general RLHF DAG) |
|---|---|---|
| Models needed per training step | Policy only (reward model runs earlier, offline, to build the preference/SFT data) | Actor, critic, reference, and reward model, every step |
| Backward passes per step | One (policy) | Two (actor and critic) |
| Resharding edges per step | None at DPO-update time (data is static preference pairs) | Many-to-many, every stage-1→stage-2 and stage-2→stage-3 transition |
| Data freshness requirement | Latest preference batches, not strictly step-by-step on-policy | Strictly on-policy per step (generation and update interleaved) |
| Meta's stated reason for choosing | — | "required less compute… and performed better" going the other way1 |
OpenRLHF, verl, and the other Ray-based RL frameworks are built to run the right-hand column of Table 2 — the harder, four-model case — because their target use cases (RLVR reasoning training with a programmatic reward, multi-turn agent RL) do not have a static, offline reward model that a DPO-style pipeline can lean on the way Llama 3's helpfulness/safety alignment does. Section 5 shows what that DAG looks like once it is mapped onto actual GPUs.
5. OpenRLHF: the DAG on real GPUs
OpenRLHF is a direct, concrete instance of Figure 3's DAG, and its own description of itself is the plainest available: OpenRLHF leverages Ray for efficient distributed scheduling. It separates the Actor, Reward, Reference, and Critic models across different GPUs, enabling scalable training for models up to 70B+ parameters
5. Each of Figure 3's four nodes gets its own dedicated pool of GPUs; Ray is the layer that knows about all four pools and moves data (and, at the actor, weights) between them, which is exactly the "coordinator for the many-to-many resharding edges" role Section 4 argued something has to play.
Within that per-role placement, OpenRLHF's architecture (its own Figure 1) defines two engine roles that cut across the four model roles: a rollout engine, built on vLLM, responsible for response generation; and a ZeRO engine, built on DeepSpeed, which computes log-probabilities (for the reference and, where needed, the actor) and runs training (for the actor and critic)4. The actor is the one model that needs both: it generates via the rollout engine and trains via the ZeRO engine, so its GPU pool runs both pieces of software, while the reward and reference models — which only ever need a forward pass, never a gradient — only need the ZeRO engine's inference path.
PagedAttention reappears at this layer, not just inside Llama 3's bespoke rejection-sampling code path. OpenRLHF's own paper describes the same mechanism, in almost the same words as Section 3's Llama 3 quote: PagedAttention significantly reduces memory waste to less than 4%… and also supports efficient memory sharing for advanced sampling methods, such as parallel sampling and beam search, reducing memory usage by up to 55%
4. That is the second independent place an inference-serving KV-cache mechanism shows up load-bearing inside a training system — once inside Meta's own hand-rolled rejection-sampling loop, once inside a general-purpose open-source RL framework's rollout engine — which is a reasonable signal that this is not a one-off trick but the default way any framework doing repeated same-prompt sampling inside a training loop now handles its KV cache.
OpenRLHF also documents an asynchronous dataflow layer on top of the per-role GPU split: rollout engines, actor engines, and remote engines operate independently and communicate via message passing, enabling immediate processing as soon as data becomes available… this design reduces idle time, improves pipeline efficiency, and enhances scalability and flexibility across large distributed GPU clusters
4. That single sentence is the seed of Module 37's entire subject — if generation and training can proceed independently and only exchange messages when data is ready, the question of exactly how stale the weights the rollout engine is generating from are allowed to get, before the trainer's gradient update is computed on data that no longer matches the current policy, becomes the framework's central design decision rather than an afterthought.
On raw numbers, OpenRLHF's own reported comparisons (against a fixed hardware and hyperparameter setup, not against every framework the field has produced) show a 3.6× per-epoch speedup over DeepSpeed-Chat running PPO on 1,024 prompts (236.8 s vs. 855.09 s) and a 3.1× speedup over TRL running GRPO on GSM8K (1,657 s vs. 5,189 s), attributed to the combination of vLLM-accelerated generation and Ray-based distributed execution4. Against verl specifically, in a long-chain-of-thought setting, the reported speedup ranges from 1.22× at 1.5B parameters to 1.68× at 14B parameters4. These numbers describe one framework's benchmark of itself against three others under one set of conditions; a systematic multi-framework comparison, including verl, SkyRL, AReaL, slime, and NeMo-RL, is a different module's job, not this one's.
6. The 80/20 problem: generation dominates wall-clock time
Every section so far has been about correctness — which model produces what, in what order, moved between which GPUs. This section is about where the clock time actually goes once the DAG is running, because that number is what determines whether synchronous, step-locked execution (generate, then train, then generate again, waiting on each other every round) is an acceptable design or a bottleneck worth architecting around.
OpenRLHF states the number plainly: RLHF training spends 80% of the time on sample generation
5. The asymmetry is the same one you already know from serving: generation is autoregressive decode, one token per sequence per forward pass, memory-bandwidth-bound and comparatively poor at using GPU compute; a training step, whether it is DPO's single backward pass or PPO's actor-plus-critic pair, is a fully batched forward-and-backward over already-known sequences, which is exactly the regime GPUs are best at. Running K=10–30 samples per prompt (Section 2.2) multiplies the decode-bound side of that asymmetry before a single gradient is computed.
That imbalance is the whole justification for not treating Figure 3's DAG as a lock-step pipeline. If four out of every five seconds of wall-clock time are spent waiting on the slowest model to finish decoding, and the fifth second is a comparatively fast, highly-parallel training step, then a scheduler that insists on strict alternation — generate a full batch, wait, train on it, wait, generate the next batch — is spending the majority of its time with the training GPUs sitting idle. Every framework's asynchronous-dataflow feature (Section 5's message-passing description of OpenRLHF's rollout/actor/remote engines is one instance) exists specifically to let the fast side of that 80/20 split stop waiting on the slow side. The moment you decouple them, though, you introduce the question that Module 37 addresses directly: the trainer may now be updating on rollouts generated by a policy that is one or more gradient steps stale, and how much staleness is tolerable before it corrupts the update is not a question this module's Llama 3/OpenRLHF pipelines had to answer, because both keep generation and training tightly coupled within a round.
7. What this module does not cover: model registries and experiment tracking
Figure 1's loop-back arrow — "best checkpoint feeds next round" — hides an operational question this module cannot answer from its source map: what system decides which checkpoint is best, where that checkpoint is stored with a stable identifier, and how six rounds of reward models, SFT models, and DPO models (plus whatever averaged variants Section 2.3 produces) are tracked well enough that a specific one can be reproduced or rolled back. Llama 3's paper names the inputs to that decision — benchmark evaluation, the quality-analysis/human-evaluation process described in Section 2.1, capability-specific checkpoint selection1 — but does not name a registry system, a checkpoint-naming scheme, or an experiment-tracking tool anywhere in its post-training section. That silence is the honest answer: Meta's release framing documents the algorithm and the data in extraordinary detail and says nothing publicly about the MLOps tooling wrapped around it.
This module does not fill that gap with a generic model-registry/experiment-tracking primer (MLflow-style registries, W&B-style run tracking, artifact lineage graphs) because doing so would misrepresent a real absence in the source material as if it were something Llama 3's own disclosure covers. If your work requires that tooling, it is a separate, well-documented subject in its own right — just not one this module's grounding extends to. What can be stated with the paper's own words is the shape of the decision the tooling would need to support: pick, at the end of each round, "the best performing checkpoint from the previous post-training iteration, or the best performing checkpoint for a particular capability"1 — a per-capability, not globally single, notion of "best," which is itself a nontrivial requirement for whatever registry a reproduction of this pipeline would need.
Where experts disagree
Two design questions run through Sections 2–5, and the sources in this module's map answer them differently depending on which workload they were built for: whether DPO or PPO-style actor-critic RL is the right algorithm for a post-training round, and whether a model's GPU footprint should be a dedicated pool or shared with other roles.
DPO is the right choice once you have a good preference signal
Meta, for Llama 3 405B's post-training1
Meta tried PPO first and switched: DPO needed less compute at 405B scale and outperformed PPO specifically on instruction-following benchmarks like IFEval. It also collapses Figure 3's four-node DAG down to one model at update time — no critic, no per-step rollout, no reference-policy forward pass during training (the reference comparison is implicit in the DPO loss's log-ratio term, computed against the same preference pairs, not a live model).
- Lower compute per round; one backward pass, not two
- No resharding edges at update time; simpler dataflow than Figure 3's general case
- Reported better on instruction-following (IFEval) at this scale1
- Needs an offline, pairwise preference signal built in advance (Sections 2.1–2.2's whole annotation-plus-rejection-sampling apparatus)
- Cannot directly express a programmatic, per-example verifiable reward — only relative preference between two fixed responses
PPO-style RL stays dominant once the reward is verifiable, not a learned preference
RLVR-focused frameworks4
OpenRLHF's own related-work framing draws the line explicitly: reinforcement learning with verifiable rewards (RLVR) uses automatically-checkable signals — mathematical verification, code execution — rather than a static preference model, and PPO remains dominant due to its stability
in that regime even as DPO is noted as more compute-efficient elsewhere4. A programmatic reward is available every round without a separate annotation-and-rejection-sampling pipeline, which is precisely the case DPO's design does not target.
- Reward signal can be recomputed fresh every rollout, no offline preference corpus required
- Online generation-and-update loop naturally incorporates new rollouts as the policy improves
- Critic's value baseline handles credit assignment across long reasoning traces
- Four models resident every step (Table 2); higher GPU footprint and resharding overhead
- Historically less stable than DPO; needs careful reward and KL-penalty tuning
Dedicated GPU pool per model role
Give the actor, critic, reference, and reward models each their own GPUs and let Ray move data between the pools. Each role can be scaled independently to match its own bottleneck, and debugging is straightforward because a GPU's role is fixed for the run's duration. OpenRLHF's own usability comparison reports this design at 8,523 lines of code, fewer than TRL or verl4.
- Simple to reason about; independent scaling per role
- Fault domains stay separated — a crash in one role's pool doesn't take down another's
- Reward/reference GPUs sit idle during the generation phase, and actor/critic GPUs sit idle during preparation, wasting reserved capacity
Colocated, shared hybrid engine across roles
HybridFlow/verl2
HybridFlow's 3D-HybridEngine explicitly targets the previous position's weakness, calling rigid model placement in multi-controller, separate-pool systems a design flaw, and instead reshards the actor model between training and generation phases on the same GPUs with zero memory redundancy, then uses a placement-optimization algorithm to map every node in the DAG onto whichever GPUs minimize idle time2.
- Higher aggregate GPU utilization; no dedicated-but-idle pools
- Reports 1.53×–20.57× throughput improvement over the multi-controller baselines it compared against2
- Requires a resharding/handoff mechanism engineered into the runtime itself, not just an orchestration layer on top
- Colocation ties roles' lifecycles together on shared hardware, weakening the fault isolation dedicated pools give you
On algorithm choice: DPO wins for exactly Llama 3's case — a helpfulness/safety alignment target where a reward model trained on human preference comparisons is a good enough proxy and cost matters at 405B scale. PPO-family actor-critic RL keeps winning wherever the reward is a program, not a learned model — RLVR reasoning and coding tasks, where a static preference model cannot express "this proof is valid" the way a verifier can. Neither source in this map runs both algorithms head-to-head on the same benchmark at matched compute, which is the experiment that would actually settle the boundary between them rather than each camp reporting results on its own preferred task.
On GPU placement: dedicated pools win when the models differ enough in size that independent scaling matters and the team values debuggability over squeezing out the last GPU-hour — which is exactly the accessibility goal OpenRLHF states for itself. Colocated hybrid engines win when GPU budget is the hard constraint and the workload's idle-pool waste (reward/reference sitting unused during generation, actor/critic sitting unused during preparation) is large enough that verl's reported double-digit throughput multiples are worth the added engineering complexity of a resharding runtime. Both are live choices in production RL frameworks today, not a settled question — Module 37 picks the thread back up from the staleness-tolerance side of the same trainer/rollout-decoupling problem.
What is changing
Two trends visible even inside this module's narrow source map point the same direction. First, the generation share of RLHF wall-clock time is climbing, not staying fixed at Section 6's 80/20 split: OpenRLHF's original framing of "80% of the time on sample generation" is, in its own later paper, revised to "over 90% of the total RLHF (or RLVR) runtime," attributed specifically to chain-of-thought workloads where models generate thousands of reasoning tokens before an answer4. Every architectural argument Section 6 makes about why synchronous, lock-step generation-then-training wastes GPU-hours only gets stronger as that ratio moves further from 50/50.
Second, the RLVR side of the "Where experts disagree" section is not a stable minority position — it is the reason recently-built frameworks default to the harder, four-model PPO-style dataflow of Section 4 rather than Llama 3's simpler DPO pipeline. Llama 3 405B's post-training, as disclosed, has no verifiable-reward stage at all1; HybridFlow and OpenRLHF are both built around the assumption that a reward is something you compute on the fly, not something you collect from humans in advance24. A future frontier post-training paper disclosing this level of detail for a reasoning-focused model, rather than Llama 3's primarily-helpfulness-and-safety alignment target, would be the natural next primary source to re-ground this module's Section 4/5 material against — and would likely show the PPO-style DAG, not the DPO loop of Figure 1, as the disclosed pipeline shape.
References
- paperLlama Team, AI @ Meta. The Llama 3 Herd of Models. arXiv:2407.21783, 2024. https://arxiv.org/abs/2407.21783. Accessed 2026-09-18. Used for: the six-round post-training pipeline, reward modeling and rejection-sampling mechanics, PagedAttention adoption and throughput figure, DPO hyperparameters and loss modifications, preference/SFT data statistics (§4.1–4.2, Tables 6–7).
- paperSheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., Wu, C. HybridFlow: A Flexible and Efficient RLHF Framework. arXiv:2409.19256; EuroSys '25. 2024/2025. https://arxiv.org/abs/2409.19256. Accessed 2026-09-18. Used for: the RLHF dataflow-as-DAG formalism, the single-controller/multi-controller placement trade-off, PPO's four-model structure (§1–2, Fig. 1–2).
- paperKwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., Stoica, I. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. arXiv:2309.06180. https://arxiv.org/abs/2309.06180. Accessed 2026-09-18. Used for: the PagedAttention mechanism and its original inference-serving throughput/memory-waste claims (Abstract).
- paperHu, J., Wu, X., Shen, W., Liu, J. K., Zhu, Z., Wang, W., Jiang, S., Wang, H., Chen, H., Chen, B., Fang, W., Xianyu, Cao, Y., Xu, H., Liu, Y. OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework. arXiv:2405.11143v6, 2025. https://arxiv.org/abs/2405.11143. Accessed 2026-09-18. Used for: OpenRLHF's rollout/ZeRO engine architecture (Fig. 1), PagedAttention memory-waste and parallel-sampling figures, asynchronous dataflow description, throughput comparisons against DeepSpeed-Chat/TRL/verl, the >90% generation-time figure for long-CoT RLVR (§1, §3.2, §4).
- repoOpenRLHF contributors. OpenRLHF README. GitHub, 2026. https://github.com/OpenRLHF/OpenRLHF. Accessed 2026-09-17. Used for: the Actor/Reward/Reference/Critic separate-GPU placement claim and the 80%-of-time-on-generation figure.