Reinforcement Learning for LLM Post-Training · Module 17

RL Environments for LLMs: Task Datasets, Harnesses, Verifiers, State, and Reward Shaping

An RL environment is the part of the training loop that is not the model: the tasks, the program through which the model acts, the state those actions change, and the verifier that turns the final state into a number. This module rebuilds the Gymnasium contract for token and tool-call actions and reads how NeMo Gym, verifiers, OpenEnv, Reasoning Gym and seven 2025-2026 technical reports construct, shape and validate environments. Afterwards you can specify an environment for a new domain (tasks, harness, verifier, state, reward) and name what to test before it feeds a gradient.

Course 50 min read 21 sources As of 2026-10-27 Prerequisites: Module 10, Module 16
The Gymnasium agent-environment loop beside the loop of an LLM agent that acts through a harness on a sandbox whose state a verifier reads at the end Gymnasium contract Agent Env action obs, reward reset() -> (obs, info) step(a) -> (obs, reward, terminated, truncated, info) terminated: task ended or failed truncated: time limit reached action: one element of a Space reward: returned at every step LLM agent in a sandbox LLM Harness Sandbox turn text obs call result Verifier final state Trainer reward ronce, at the end tokens, logprobs action: one model turn, many tokens state: in the sandbox, not the context reward: read from the final state
Figure 1. Left, the Gymnasium contract1. Right, the loop a trainer runs for an LLM agent, a schematic drawn from TRL's environment_factory (the trainer drives generation and tool calls; the reward is read from the environment after the episode)2, τ-bench's hidden database state and LLM-simulated user3, and the verifiers library's separation of harness, runtime and scoring4. The orange arrows have no Gymnasium counterpart: scoring happens once, from state the agent never sees directly.

1. From reset and step to a trajectory of turns

1.1 What the Gymnasium contract promises

The 2016 OpenAI Gym whitepaper fixed the shape every environment library since has copied: env.reset() returns the first observation, env.step(a) returns (observation, reward, done, info), and the library abstracts the environment only, not the agent. It also set a rule that matters for LLM work: any change to an environment's behavior must come with a new version name, because results before and after are incomparable5. Gymnasium split done into two flags in v0.26 "to make it clearer to users when the environment had terminated or truncated", which it calls "critical for reinforcement learning bootstrapping algorithms". terminated means the agent reached a terminal state of the task's MDP, positive or negative; truncated means a condition outside the MDP held, typically a time limit; either requires a reset()6. reset returns (observation, info) and takes seed and options; info may hold "individual reward terms that are combined to produce the total reward"6. The docs call Env an MDP "not a perfect reconstruction, missing several components"1.

The contract carries four assumptions that were free in Atari and MuJoCo and are not free for an LLM: the environment owns dynamics and reward; the action space is declared up front as a Space (Box, Discrete, Dict, and a Text space for strings with minimum and maximum length1); one step takes one action and returns one reward; and stepping is cheap. An agent that emits thousands of tokens per action on a machine that must be built, snapshotted and graded violates the last two (my reading; the docs make no such claim).

1.2 What the LLM does to each element

Module 2 gave two ways to cast a single completion into the contract: a token-level MDP and a bandit whose single action is the whole completion. A tool-using agent adds a third unit, the turn. A trajectory is \(\tau=(x,a_1,o_1,a_2,o_2,\dots,a_K)\): each \(a_k\) is a token sequence sampled from \(\pi_\theta\) (reasoning plus a tool call or final answer) and each \(o_k\) is text the environment wrote. The log-probability splits into a policy part and an environment part, $$\log p_\theta(\tau)=\sum_{k}\log\pi_\theta(a_k\mid x,a_{\lt k},o_{\lt k})+\sum_{k}\log P(o_k\mid x,a_{\le k},o_{\lt k}),$$ and only the first sum depends on \(\theta\). So a unit test, a browser or a simulated user can sit in the loop without being differentiable, as Module 2 argued for a single completion. Not every token in \(\tau\) is a policy token, though: the \(o_k\) segments sit in the context but were not sampled, which is where loss masks come from (Module 18).

Table 1. The Gymnasium elements and what they become for an LLM. The single-completion column follows Module 2; the multi-turn column follows TRL's environment_factory contract2 and τ-bench's formalization3.
ElementSingle completionMulti-turn tool use
observationprompt, then tokens so faruser message, tool output, page text; history re-enters the context
actionnext token, or whole completionone model turn: reasoning plus a parsed tool call or answer
action spacethe vocabularythe tool schema: names and typed arguments
resetsample a promptload a task, build or restore the sandbox, return the first observation
stepappend a tokenexecute the call in the sandbox, return its output as text
rewardreward-model score at EOS, or verdictverifier on final state; sometimes per-step terms
terminatedEOSmodel submits; task solved or failed
truncatedmax_new_tokenscontext cap, turn cap, wall-clock timeout
hidden statenonefiles, database rows, processes, browser session

Two rows deserve a closer look. The action space is now a schema, not a vocabulary. TRL discovers every public method of the environment class, exposes it as a function-calling tool, requires typed docstrings to build the schema, and advises against a generic step(action) because "the model needs meaningful tool names and argument descriptions to learn tool calling"2. And the state is no longer the observation. τ-bench writes its environment as a POMDP with \(S=S_{db}\otimes S_{user}\), \(A=A_{db}\cup A_{user}\), \(O=O_{db}\cup O_{user}\) and reward \(R:S\to[0,1]\); the database is "hidden from the agent and the user" and reachable only through API actions, while the user is a language model (gpt-4-0613) whose state is its prompt plus the conversation3. An environment can contain a model. The context window is a log of observations standing in for a belief state; the true state is in the sandbox, where the verifier reads it.

1.3 Terminated, truncated, and what to do with a cut-off episode

Gymnasium's split exists so that a bootstrapping learner can treat a time-limit cut differently from a real end6 (Module 2 explains why for a critic). A critic-free recipe faces a different question: should a truncated trajectory be graded at all? The sources document three answers. Grade it as usual, the default when the verifier reads whatever state exists. Skip it: verifiers has skip_on_truncated and skip_on_timeout switches that leave a cut-off run with no rewards, so its reward is 0 and trace.info["scoring_skipped"] records why4. Or penalize it: Kimi K2 enforces a per-sample token budget by task type, and "responses that exceed this token budget are truncated and assigned a penalty"7 (§3.2.3). TRL adds a trap for multi-turn work: max_completion_length caps tokens across the whole conversation, "all model generations + tool results combined", so verbose tool output can exhaust it2. No source states a consensus on which flag to set or what reward should follow.

class SqlEnv:
    """One instance per rollout. State lives in the sandbox, not in the context."""

    def reset(self, task):
        self.db = sandbox.restore(task["snapshot"])        # task = one dataset row
        self.gold, self.turns = task["gold_rows"], 0
        return task["question"], {"task_id": task["id"]}    # (observation, info)

    def step(self, action):                                # action = one parsed model turn
        self.turns += 1
        if action.tool == "run_sql":                       # exploration: no reward yet
            obs = self.db.execute(action.args["query"])    # tool output returns as text
            return str(obs), 0.0, False, self.turns >= 8, {}
        if action.tool == "submit":                        # final answer, graded from state
            rows = self.db.execute(action.args["query"])
            r = float(sorted(rows) == sorted(self.gold))   # unordered result-set equality
            return "", r, True, False, {"verifier": "result_set_equality"}
        return "unknown tool", 0.0, False, self.turns >= 8, {}

The contract as a sketch (mine, not any library's API). The reward rule mirrors the Bird SQL verifier in NeMo Gym's catalogue (Section 4.2). Everything hard is hidden in sandbox.restore, execute and the dataset; the rest of this module is about those three.

2. Anatomy: tasks, harness, state, verifier

2.1 Four parts under four vocabularies

NVIDIA's NeMo Gym defines an environment as "the complete system an agent interacts with to complete a task": a dataset (the tasks), an agent harness ("how the model interacts with the world"), a verifier and per-task state. Its quick start starts three servers, a mcqa resources server, a mcqa_simple_agent agent and a policy_model, so, as far as the README shows, tasks and verifier live in the resources server, the harness in the agent server and the model in a third (my reading of the layout). The README is candid about overkill: "if you're scoring model outputs with a stateless check and don't need scale or training, a script is probably sufficient"8.

The verifiers library cuts the same object differently. A Taskset loads Tasks from a dataset or generator; a Task holds the lifecycle hooks (setup, finalize), tools, stop conditions, metrics and rewards; TaskData is the immutable record of prompt, reference answer, artifacts and per-task limits; the agent solves it "inside a harness running in a runtime" (subprocess, Docker, remote sandbox or custom)4. OpenEnv cuts at the network boundary: each environment is a Docker image exposing reset(), step() and state(), sync or async, over a WebSocket9. Table 2 lines them up; the rows for NeMo Gym, verifiers and OpenEnv are snapshots of main-branch documents and will drift.

Table 2. Environment interfaces compared, read from each project's own documentation or README on 2026-10-27; none is pinned to a release except where stated.
LibraryUnitInterfaceIsolationNote
Gymnasium6Env classreset, five-element stepin-processsingle-agent
OpenEnv9Docker image plus clientreset, step, state over WebSocket; MCP at /mcpcontainer per environment"early development"; 40+ environments
NeMo Gym8dataset + harness + verifier + stateresources, agent and model serversseven sandbox providers by v0.5.0README shows v0.7.0 (10/06/2026)
verifiers4Taskset of Tasks in a runtimeasync Python; @vf.reward(weight), @vf.metric, built-in judgessubprocess, Docker, remote; optional isolated scoringdocs on main, no version
Harbor10task directoryharbor run with an "oracle" agentDocker, Daytona, Modaltutorial shows rewards 0 and 1
TRL environment_factory2Python class per generationreset(**row); public methods become toolsnone by itselfOpenEnv servers default to 1 session

2.2 One rollout as a lifecycle

Whatever the vocabulary, a rollout passes through the same stages, and the order is where environments go wrong. The verifiers documentation states the lifecycle most explicitly, and the stepper below follows it.

Load task Setup Agent loop Finalize Score Emit trace TaskData image + files turns, tools saved state rewards tokens + r sandbox state: files, database rows, processes (exists from setup to scoring) Load task TaskData: prompt, reference answer, artifacts, limitsfrom a dataset or an (optionally infinite) generator Setup Task setup: files, repositories, services in the sandboxthe harness is installed in the same runtime Agent loop Model and tools run until done or a stop conditionmodel traffic passes an interception server that records the trace Finalize Capture state before grading mutates itexample: save the agent's git diff before test files are checked out Score Metrics, then weighted rewards, then judgesa failing judge raises an error; it does not score 0 Emit trace Trace to the trainer: messages, rewards, exact token idstoken ids are captured where the model is called
Figure 2. The rollout lifecycle. Stage 1 follows the verifiers taskset loader and stages 2 to 5 its documented lifecycle (task setup, harness setup, agent execution, finalize, scoring)4; stage 6 follows NeMo Gym's release notes on token-level trajectory capture8 and OpenEnv's Harbor integration, which "captures the exact tokens for RL"9.
  1. Load the task. The taskset yields tasks from a dataset or generator; the client ships each TaskData to the worker that runs it. A generator can declare itself infinite, and every run must then bound it4.
  2. Set up. Task setup prepares files, repositories or services in the sandbox, and the harness is installed in the same runtime4. Harbor builds the image from environment/Dockerfile, allowing 600 s in its tutorial example10.
  3. Run the agent loop. The model and tools run until completion or a stop condition. verifiers routes model traffic through an interception server that builds traces live, which "prevents agents tampering with their own traces", and can run stop hooks that rewrite tool responses "for example, to block reward hacks"4.
  4. Finalize. finalize "captures any outputs or state needed for scoring and inspection before tests mutate the workspace"; for software tasks it saves the agent's git diff first. A run stopped mid-turn has its harness closed before grading, so the agent cannot edit during it4.
  5. Score. Metrics run first and affect nothing; rewards sum as score x weight (weight 0 logs a signal without training on it); judges run last. If a judge fails, the documentation says to raise "instead of returning 0.0, which would record an infrastructure failure as a wrong answer by the model"4.
  6. Emit the trace. The trainer receives messages, rewards, metrics and exact token ids captured at the model call, not re-tokenized from text. NeMo Gym v0.6.0 describes preserving "exact token IDs across multi-step runs" with external harnesses, and v0.7.0 adds durable worker-side capture for vLLM and Megatron8. What the loss does with them is Module 18.

2.3 A concrete spec: the Harbor task directory

The cleanest statement of what a terminal-style task must contain is Harbor's tutorial, the format GLM-5's terminal environments use (Section 6.3). A task is a directory:

ssh-key-pair/
|-- instruction.md          what the agent receives
|-- task.toml               metadata; [verifier] timeout_sec = 120.0, [agent] timeout_sec = 120.0,
|                           [environment] build_timeout_sec = 600.0
|-- environment/Dockerfile  the container the agent works in
|-- solution/solve.sh       reference solution, run by the "oracle" agent
`-- tests/
    |-- test.sh             verifier entry point; must write /logs/verifier/reward.txt
    `-- test_outputs.py     pytest checks run by test.sh

The tutorial's test.sh writes 1 to the reward file if pytest exits 0 and 0 otherwise; for "multi-criterion verifiers, weighted scoring, or LLM-as-a-judge rubrics" it points to a separate library, RewardKit10. The validity check is built in: harbor run -p ssh-key-pair -a oracle executes solve.sh and a reward of 1 is the pass condition10. A task whose reference solution does not score 1 is broken whatever a policy later does with it; every pipeline in Section 6 contains this check under another name.

2.4 A census of one catalogue

The NeMo Gym README's environment table is the largest public list of RL environments among the sources. As of the README fetched on 2026-10-27 it has 226 rows under 152 distinct names; repeated names are variants (Reasoning Gym appears nine times, the SWE-bench agent seven, single-step tool use seven) (my count; Figure 3). Agentic, coding and knowledge rows dominate, and 202 of the 226 rows show "-" in the Dataset column, which the README defines as data "not publicly released yet" or "procedurally generated using a provided script"8. The environment code is public and most of the environment data is not (derived: 24 rows link a public dataset). The README's "battle-tested in production Nemotron training" is a vendor claim; Nemotron 3 Ultra's report says it open-sources recipes, data and RL environments with the checkpoints11. The Nemotron repository README (fetched 2026-10-27) is narrower: its Ultra recipe "covers Pretrain → SFT", trains "exclusively on the open-sourced subset of training data", and states "Results will differ from the tech report benchmarks, which used additional proprietary data"12. RL environments may sit in the NeMo Gym or NeMo RL repositories; I inspected neither beyond the NeMo Gym README (Module 32).

Rows of the NeMo Gym environment table by domain label: agent 45, knowledge 36, coding 33, blank 28, math 25, other 23, instruction following 16, rlhf 7, games 6, safety 5, long context 1, e2e 1 010203040 rows in the table agent45 knowledge36 coding33 (domain blank)28 math25 other23 instruction_following16 rlhf7 games6 safety5 long_context1 e2e1
Figure 3. Rows of the NeMo Gym README environment table by Domain label, 226 in total (my count, README fetched 2026-10-27)8. The orange bar is rows with no label; their names (SWE-Gym, SWE-rebench, R2E-Gym and variants) suggest software engineering. A row is a configuration, not a distinct environment: there are 152 distinct names.

3. The harness: who owns the loop

3.1 White-box loops and loop-owning harnesses

The harness is the program between the model and the sandbox: it renders the prompt, parses tool calls, runs them and decides what stays in context. OpenEnv's README names two ways to put one in a training run. In "environments as tools (white-box)" the trainer runs the multi-turn loop and the environment supplies tools and reward. In "real agent harnesses (loop-owning)" the agent runs its own loop and the integration "captures every model call as a framework-neutral TrainingTrace"9. TRL's guide draws the same line, calling its environment_factory path white-box and keeping rollout_func for external agent servers that own the loop2. Kimi K2.5 has built-in white-box environments and black-box ones "that can only run under standard LLM API protocol"13 (Appendix D).

The black-box case needs more than text: the trainer needs the exact token ids the model sampled (Module 18 covers why). Five systems solve it at the model boundary rather than inside the harness (Table 3); three of them describe the device as a proxy or gateway that speaks the harness's native API and records tokens in passing. verifiers' interception server forwards "using the API surface that the harness expects, such as OpenAI Responses for Codex or Anthropic Messages for Claude Code"4; NeMo Gym lists Claude Code and Codex against a model server's /v1/messages and /v1/responses8; K2.5's LLM Gateway "keeps detailed records of rollout requests and responses"13; GLM-5's TITO gateway "intercepts all generation requests from rollout tasks and records each trajectory's token IDs and metadata"14 (§4.1.2). System Design Modules 27 and 37 cover the harness, sandbox and serving engineering.

Table 3. How each system lets a harness it does not control feed a trainer, from each source's own text; the sources do not benchmark these against one another.
SystemTrainer-run loopHarness-run loopWhere tokens are recorded
OpenEnv + TRL92environment_factoryHarbor integration: Claude Code, Codex, OpenCodeTrainingTrace per model call
verifiers4harness in the runtimeany harness with a provider APIinterception server, live traces
NeMo Gym8in-tree agents (OpenHands, LangGraph)Claude Code, Codex via /v1/messages, /v1/responsesexact token ids (v0.6.0); worker-side capture (v0.7.0)
Kimi K2.513built-in white-box environmentsblack-box via LLM Gatewaygateway logs; token-in-token-out
GLM-514rollout logic in the frameworkexternal agent frameworksTITO gateway, per trajectory

3.2 Harness diversity as a training-data property

Two reports treat the harness as something to vary. Kimi K3: "training with a single fixed agent harness can cause a model to overfit to a particular tool schema, system prompt, context management mechanism, or interaction protocol". Its white-box environment represents a harness as "a collection of configurable, composable modules" (tools, system prompts, context strategies, skills, memories, subagents), instantiates Kimi Code, Claude Code, Codex, OpenClaw and Hermes "as well as entirely new ones", and varies configurations by task group15 (§4.2.1); web-development tasks use "diverse agent scaffolds rather than a single fixed harness"15 (§4.2.7). Nemotron 3 Ultra sorts its tasks into five verticals and trains each under at least two of six harness families (Stirrup, OpenHands, OpenCode, Terminus, Droid, internal), "reducing overfitting to any particular harness design"11 (§3.2, Appendix A.2). NeMo Gym shows the mechanism in configuration: the same Reasoning Gym tasks pair with Claude Code, Codex, and LangGraph orchestrator, parallel-thinking, reflection and ReWOO agents8. With the harness a config axis, diversity costs a YAML file instead of a new environment.

The harness changes the task itself. DeepSeek-V3.2 keeps historical reasoning when only tool outputs are appended and discards it when a new user message arrives; frameworks that simulate tool results as user messages, such as Roo Code or Terminus, "may not fully benefit", so the report recommends non-thinking mode for them16 (§3.2.1). One model under two context policies solves two different sequential problems, which argues for diversity and for pinning harness versions in reports. Neither K3 nor Nemotron states a harness version in the sections read.

3.3 Environments that are not interactive

Not every agent environment runs a sandbox end to end. NeMo Gym's "PivotRL" verifiers score one proposed tool call against a demonstrated action: a single-step search verifier requires queries to match a golden word bag exactly, applies list-F1 guardrails, and scores a response with more than one tool call as 0, aiming at "eliminating partial-credit reward hacking"8. Nemotron 3 Ultra reports "single-turn rollouts, similar to PivotRL" for most agentic tasks in its distillation stage11 (§3.3; Module 18 gives its reason). The trade is in the catalogue's own phrase, "behavior cloning per step": no sandbox cost or long-horizon credit assignment, but the target becomes matching a demonstration8.

4. The verifier reads state, and the reward has a shape

Module 10 covers how to build a verifier for an answer string, a test suite or a constraint, and Module 16 covers judges. This section covers what is specific to an environment that has state: what the verifier is allowed to read, which reward shapes the shipped environments use, what shaping theory says about dense rewards, and the gates reports build into verifiers.

4.1 Grade the state, not the transcript

τ-bench is the cleanest worked example of a state-based reward. Each task has a hidden annotation of the ground-truth database writes; the simulated user's instruction is written so "only one possible outcome" exists under the domain policy; and the reward is a product of two binary checks, \(r=r_{\text{action}}\times r_{\text{output}}\in\{0,1\}\): the final database must equal the unique ground-truth database, and the agent's replies must contain the required strings. Dialogue and read-only calls can vary freely3. The paper's example succeeds if the only write is one specific return_delivered_order_items call and the messages contain "54.04" and "41.64". The benchmark has 115 retail tasks (500 users, 50 products, 1,000 orders) and 50 airline tasks (500 users, 300 flights, 2,000 reservations)3 (Table 1). The authors flag the residual false positive: \(r=1\) "might be a necessary but not sufficient condition", since the agent could issue the return without explicit user confirmation3. Reading state removes transcript ambiguity and leaves the policy-compliance gap.

Kimi K3's Autonomous Execution Tasks state the principle in one line: "rewards are grounded in the verifier's evaluation of the final environment state rather than the agent's self-reported completion"15 (§4.2.6). TRL's reward-design tips say the same in practice: "check the final state, not the path", so the model is free to discover its own strategies2.

4.2 A catalogue of reward shapes

Module 10 (Section 3.4) reads NeMo Gym's Bird SQL, Spider2, Blackjack, GUI-coordinate and single-step tool-use verifiers. Table 4 adds the other shapes that appear in the shipped environments and the reports, and Figure 4 draws the shape classes. The plots are schematics: where a source states only the shape in words, I draw that shape and say so.

Six reward shapes as small plots: ternary, binary equality, quadratic distance, gated then performance-scaled, milestone with a veto, and dense per-step sum Sparse ternary lossdrawwin -10+1 Blackjack Binary equality differequal 01 Bird SQL result sets Quadratic distance 10 distance to target GUI grounding (shape only) Gate, then scale ? 10.50 wrongexpertroofline K3 kernels (schematic) Milestones with veto minefield: 0 10 milestones reached ToolSandbox (schematic) Dense, summed return = sum of step rewards step 1 ... 6 OpenAir congestion (schematic)
Figure 4. Reward shapes in shipped environments. The ternary and binary values are stated in the NeMo Gym README; the quadratic, milestone and per-step panels draw shapes it gives only in words8. In the kernel panel, 0 (error above threshold), 0.5 (expert parity) and "toward 1" (near the hardware roofline) are Kimi K3's; the reward below expert parity is not stated (marked "?") and the slope above it is an assumption15 (§4.2.4).
Table 4. Reward shapes used by shipped environments and technical reports. "Shape" is my classification of the source's wording.
EnvironmentReward as describedShape
Blackjack8+1 win, 0 draw, -1 loss or bustsparse ternary
Bird SQL, Spider2 Lite8binary, from unordered result-set equality; no LLM judgebinary on state
GUI coordinate8"smooth quadratic reward based on Euclidean distance"smooth, distance-based
ToolSandbox8milestone similarity in [0, 1]; 0 if a minefield is hitgraded with a veto
OpenAir congestion8dense per-step KPI reward; the agent sums step rewards into the episode returndense
Long-form MT (LongMT)8reference-free COMETKiwi score of a document-level translationlearned metric as verifier
Abstention8three-tier reward with an LLM judge; rewards abstaining over answering wrongly (Module 24)tiered
τ-bench3\(r_{\text{action}}\times r_{\text{output}}\in\{0,1\}\): database equals ground truth and replies contain required stringsproduct of binary state checks
K3 kernel tasks150 above a numerical-error threshold; 0.5 at expert performance, rising toward 1 near the hardware roofline; hack penaltiesgated, then performance-scaled

4.3 What shaping theory says about dense rewards

Ng, Harada and Russell's 1999 result is why dense rewards draw suspicion. Their Theorem 1 says a shaping reward \(F\) preserves optimal policies, without prior knowledge of the MDP, if and only if it is potential-based, \(F(s,a,s')=\gamma\Phi(s')-\Phi(s)\)17 (§3). The paper opens with two bugs from non-potential shaping: a bicycle agent rewarded for moving toward the goal learned to ride in tiny circles, and a soccer robot rewarded for touching the ball learned to stay beside it and "vibrate"17 (§1).

With \(\gamma=1\), the usual setting for LLM episodes (Module 2), a derivation of mine shows what a potential does: the shaping terms telescope, $$\sum_{t=0}^{T-1}F(s_t,a_t,s_{t+1})=\Phi(s_T)-\Phi(s_0).$$ A potential-based dense reward is therefore a function of the final state (up to the constant \(\Phi(s_0)\) for a fixed start) added to the sequence return, and for a group-relative method only the differences of \(\Phi(s_T)\) across the group survive. A safe dense reward carries no more path information than a terminal reward on a final-state score \(\Phi\), so the design question is the choice of \(\Phi\): partial credit by another name. A milestone count credited whenever it is reached is not a potential and can be farmed by revisiting, the cycle argument of the 1999 paper.

Partial credit on the final output is a different object: another terminal reward, whose maximizer matches the original when the top score is reachable only by a correct output. It changes the ordering among failures, which binary treats as equal. TRL reports that in Wordle and Sudoku "binary rewards (1.0 for success, 0.0 otherwise) gave cleaner training signals than shaped rewards with partial credit", arguing that "GRPO compares completions within a group, so the relative ranking matters more than the absolute values", but gives no numbers2. Module 10 (Section 4) works the group arithmetic.

4.4 Gates and tamper resistance built into the verifier

Module 19 owns the exploit taxonomy; this lists the designs. K3's kernel tasks zero the reward above a numerical-error threshold, score performance against an expert implementation, and run "a hacking-detection system that penalizes reward-hacking strategies such as CUDA graph replay, input caching, and precision reduction", extended "as new hacking strategies are observed"15 (§4.2.4). The reward function therefore changed during the run; Gym's versioning rule (Section 1.1) would call that a new environment, and the report does not say how versions were tracked (my reading). K3's Autonomous Execution Tasks isolate agents from verifiers, pair "public verifiers that offer diagnostic feedback with hidden verifiers that evaluate held-out scenarios", and use "penalty-based rewards under limited submission budgets"15 (§4.2.6). verifiers offers isolated scoring: artifacts restored into a fresh container, retried up to 2 times by default on retryable errors4. GLM-5's terminal pipeline has a refine agent check that "the environments are robust against potential exploits or shortcuts"14 (§4.2.2).

5. Procedural environments: a generator, a verifier and a difficulty knob

5.1 An environment as a distribution, not a list

Reasoning Gym (Stojanovski et al., NeurIPS 2025 Datasets and Benchmarks) replaces the fixed dataset with a program: "over 100 data generators and verifiers" across algebra, arithmetic, computation, cognition, geometry, graph theory, logic and games, with "the ability to generate virtually infinite training data with adjustable complexity"18. Its principles: every task algorithmically verifiable (P1); large solution spaces, "rewarding generalizable strategies above overfitting and mitigating reward hacking" (P2); and "configurable parameters systematically control problem characteristics, enabling dynamic curricula via precise difficulty adjustment" (P3)18 (§2). Parameters come as difficulty (node counts, polynomial degrees, word lengths), structural (dimensionality, constraint types, proof depth) and stylistic (variable names, number formats, framing), the last varying presentation without changing difficulty18. verifiers has the same pattern as a taskset whose load() is a generator with INFINITE = True, which forces every run to bound it4; NeMo Gym wraps Reasoning Gym as a resources server8.

The environment is thus a distribution \(x\sim\mathcal D(\kappa)\) indexed by a configuration \(\kappa\) that the loop can move while it runs. Module 10 (Section 5) says why that matters: a prompt yields a gradient only when a group mixes successes and failures, so the useful band of \(\kappa\) is where the pass rate is neither near 0 nor near 1.

5.2 The cliff and the curriculum

Two measurements in the paper locate that band. On hard configurations the zero-shot ranking separates reasoning models (o3-mini 63.5%, DeepSeek-R1 59.5%, Grok 3 Mini 55.1%) from general-purpose ones (Llama 4 Maverick 41.5%, Claude 3.5 Sonnet 40.3%, Gemma 3 27B 20.3%)18 (§3). And moving from easy to hard parameter settings produces what the authors call a difficulty cliff: DeepSeek-R1's score falls by 61.8 on code, 29.6 on graphs, 27.9 on algorithms and 11.8 on geometry; o3-mini's by 71.9, 33.8, 25.6 and 33.118 (the paper's Figure 3b, reported as "Δ score (%)").

The curriculum experiment trains Qwen2.5-3B-Instruct with GRPO under two regimes. The curriculum starts at the easiest level and advances "when performance exceeds 70% over 20 training steps"; the baseline samples uniformly from all levels; both train for one epoch and are evaluated on 50 held-out examples per level18 (§5). Figure 5 plots two of the three environments in the paper's Table 5; Table 5 below reproduces all three.

Accuracy by difficulty level for the base model, uniform-sampling GRPO and curriculum GRPO on Spell Backwards and Mini Sudoku base model uniform sampling curriculum Spell Backwards 20 40 60 80 0 70.7 4 30.0 6 3.4 8 0.0 10 word length (letters) Mini Sudoku 20 40 60 80 0 56.0 4-6 28.0 6-8 20.0 8-10 5.3 10-12 empty cells
Figure 5. Held-out accuracy by difficulty level, Qwen2.5-3B-Instruct, one epoch of GRPO: base model, uniform sampling over levels, and a curriculum that advances at 70% accuracy over 20 steps. Labels give the curriculum value. Data: Reasoning Gym Table 518. The base model's own collapse with word length (12.0, 3.3, 0.0, 0.0) is the cliff; the curriculum moves accuracy at the easy end and leaves the hardest level where it was.
Table 5. Reasoning Gym curriculum results, accuracy in % on 50 held-out problems per level (Table 5 of the paper18). The last column is curriculum minus uniform sampling, in percentage points.
Environment (level = …)LevelBaseUniformCurriculumGain (pp)
Spell Backwards (word length)412.0030.0070.67+40.67
63.3310.6730.00+19.33
80.001.133.37+2.24
100.000.010.01+0.00
Mini Sudoku (empty cells)4-61.1354.0056.00+2.00
6-80.0025.3328.00+2.67
8-100.006.6720.00+13.33
10-120.001.135.33+4.20
Count Primes (number range)100-50012.004.0030.67+26.67
100-10003.335.0312.67+7.64
100-50001.330.003.03+3.03

Three things follow. The gain is largest where the base model is mid-range and vanishes at the cliff: Spell Backwards at length 10 is 0.01% under both regimes, so one epoch of curriculum does not cross it. The curves show the authors' mechanism: "increases in difficulty level are followed by a sharp drop in reward", and in Count Primes the curriculum model "never advanced past the initial level" yet beat uniform sampling at every level18 (the paper's §5 and its Figure 6). My reading, consistent with Module 10's zero-variance argument and not stated by the paper, is that uniform sampling spends part of each batch where the pass rate is near zero and groups carry no gradient. The authors also caution that a curriculum's value "may be more limited in environments where progression paths are ambiguous or difficult to formalize"18: a difficulty knob is a property of the domain, and most agentic domains lack one.

Other results bound the pattern. RLVR on one category transfers within it, and an RG-Math model trained on Qwen2.5-3B-Instruct gains 9.7 on MATH, 7.7 on Big-Bench Hard and 0.5 on GSM8K (the paper's "%"), at "approximately 1500 A6000 hours" for all experiments18. The authors list limits: knowledge- or creativity-heavy domains resist generators; verification functions "may not capture all aspects of solution quality"; and the library "focuses on single-turn, text-based reasoning"18 (§7). Module 20 covers how to read transfer claims.

6. Building environments at scale

Environment construction is the least documented part of the stack in the comparative sense: every report describes its pipeline, almost none ablates it, and the numbers that exist are scale counts rather than quality measurements. Table 6 collects each pipeline's source, validity gate and difficulty filter; the text reads them in order of how much each discloses.

6.1 The funnel, priced

SWE-bench fixed the template for turning repositories into verifiable tasks. It keeps merged pull requests that resolve an issue and edit test files, runs the PR's tests before and after its other changes, and keeps an instance only if at least one test flips from fail to pass and installation succeeds; it drops tasks whose tests call names first introduced by the solution, since resolving those "may be an impossible task even for human developers"19 (§2.1, Appendix A.3). Its Table 10 funnel runs from 93,139 crawled pull requests through 11,407 task instances to 2,294 validated ones: yields of 2.46% and 20.1% (derived). A patch resolves an issue only if it applies and all the task's tests pass19 (§2.2).

SWE-Gym, the first training-oriented version, says why the validated fraction is the scarce part: its 64,689 raw instances over 358 repositories "do not contain executable environments or a guarantee that [their] unit tests are effective", so the authors "semi-manually create executable environments" for 11 repositories20 (§3.1). Validation (the gold patch must pass more unit tests than the original code) "required approximately 200 human annotation hours and 10,000 CPU core hours" for 2,438 instances shipped as 6 TB of Docker images: about 4.1 CPU core-hours, 4.9 human-minutes and, if TB means 1012 bytes, 2.5 GB per instance (derived). Its repositories are disjoint from SWE-bench's to avoid contamination20.

6.2 Replacing the human hours with a setup agent

The 2025-2026 reports automate that step. DeepSeek-V3.2 mines "millions" of issue-PR pairs, filters them by heuristics and LLM judgments, and has a V3.2-powered setup agent install packages, resolve dependencies and run tests, with JUnit output so parsing is uniform. The gate is stated precisely: "an environment is deemed successfully built only when applying the gold patch results in a non-zero count of false-to-positive (F2P) test cases … and a zero count of pass-to-fail (P2F) test cases (indicating no regressions)"; the result is "tens of thousands" of environments across languages that include Python, Java, JavaScript, TypeScript, C, C++, Go and PHP16 (§3.2.3). GLM-5 uses the RepoLaunch framework and LLM-written log parsers to extract fail-to-pass and pass-to-pass tests, and reports over 10,000 environments across thousands of repositories and nine languages14 (§4.2.1). Neither gives a funnel yield, so SWE-bench's 2.46% cannot be compared with the 2026 pipelines.

6.3 Environments with no repository to start from

Most agentic domains have no pull-request history. Four reports show four ways to manufacture the tuple of environment, task and verifier.

DeepSeek-V3.2, general agents. A synthesis agent gets a task category and a sandbox with bash and search, fetches data into a sandbox database, writes task-specific tools as functions, then proposes a simple task with a Python solution function and a Python verification function. The solution may call only the tools or do logic, "and cannot call other functions or directly access the database, ensuring the task can only be solved through the tool interface"; its output must pass the verifier, and the agent edits either until it does. It then raises the difficulty, augmenting the toolset when needed (Figure 6). The target is tasks that are "hard to solve but easy to verify", illustrated by a trip-planning task with a 15-function tool set and a constraint check on the submitted itinerary16 (§3.2.3).

DeepSeek-V3.2 general-agent environment synthesis: fetch data, write tools, propose a task with solution and verifier, validate, raise difficulty in a loop, then filter by pass@100 to 1,827 environments and 4,417 tasks Category + sandboxbash and search tools Fetch datainto a sandbox database Write task toolseach tool a function Propose tasksolution fn + verifier fn Solution passes verifier?solution uses tools only Raise difficultyadd tools if needed yes no: edit either function harder task, updated solution and verifier Thousands of tuples(env, tools, task, verifier) RL with V3.2keep pass@100 > 0 1,827 environments4,417 tasks when the task is hard enough
Figure 6. DeepSeek-V3.2's synthesis of general-agent environments, redrawn from the workflow in §3.2.3 of the report16. The orange path is the difficulty loop; the grey path is the final filter. The report does not say when the loop stops ("iteratively increases the difficulty"), so the stopping rule in the lower left is my gloss.

GLM-5, terminal and search. Terminal tasks come from two pipelines. From seed tasks, an LLM drafts verifiable tasks, a construction agent writes them in Harbor format, and a refine agent iterates against manual rubrics so images build, tests match the task text and shortcuts fail: "thousands" of environments with "Docker construction accuracy exceeding 90%". From web pages, a coding agent writes a task and runs the Harbor validation script on its own output until it passes; "only tasks that successfully clear this self-verification loop are admitted"14 (§4.2.2). Such a loop shows a task is internally consistent, not that it is worth learning from (my reading). Search questions come from a web knowledge graph over more than two million pages, then a three-stage filter: drop a question if a tool-free reasoning model answers it correctly in at least one of eight attempts; drop it if an early-stage agent solves it in a few steps; have a verification agent validate question and answer "bidirectionally", rejecting non-unique answers14 (§4.2.3).

Kimi K3. Agents grow a hierarchical knowledge graph by web search; sampled nodes drive retrieval of public materials, and a synthesis agent writes tasks of a chosen type. Personal-assistant tasks run in mock Gmail, Notion, Slack and Canvas that "preserve the core semantics of their real-world counterparts", over multiple simulated days with dozens of interdependent events, each with its own rule or LLM criterion; "a single rollout may involve up to thousands of tool calls and millions of context tokens". Autonomous Execution Tasks specify an initial state, a constrained goal, a tool-based action space, budgets and an independent verifier, and show the agent no "reference trajectories or predefined procedures"15 (§4.2.2, §4.2.5, §4.2.6).

Kimi K2: a correction to the source map. K2's pipeline (3,000+ real MCP tools, 20,000+ synthetic tools evolved from domain hierarchies, thousands of agents, a rubric per task, LLM-simulated users, and a tool simulator "functionally equivalent to a world model" that keeps state) sits in the report's supervised fine-tuning section. It produces demonstrations: an LLM judge keeps only trajectories that meet the rubric, "effectively implement[ing] large-scale rejection sampling", with real sandboxes where "authenticity is crucial"7 (§3.1.1). The RL section describes something else: a Gym-like framework, many verifiable tasks, a Kubernetes sandbox service with "over 10,000 concurrent sandbox instances", and a self-critic reward7 (§3.2). The report does not say the simulated environments are reused for RL, and this module does not assume it.

Table 6. Environment construction pipelines as described in their sources. "Not stated" means the sections read do not say; it does not mean the step is absent.
PipelineSource materialValidity gateDifficulty filterReported scale
SWE-bench19PRs from 12 Python repositoriesat least one fail-to-pass test; install and runtime succeednone (benchmark)93,139 to 11,407 to 2,294
SWE-Gym20PRs from 11 Python repositoriesgold patch passes more tests; semi-manual setupnot stated2,438 validated; 64,689 raw
V3.2 code agent16millions of issue-PR pairsgold patch gives F2P > 0 and P2F = 0not stated"tens of thousands" of environments
GLM-5 SWE14issue-PR pairs, RepoLaunch setupF2P and P2P tests extracted by LLM-written log parsersnot statedover 10k environments, 9 languages
V3.2 general agent16LLM-written data, tools, task, verifiertools-only solution must pass the verifieriterated harder; keep pass@100 > 01,827 environments; 4,417 tasks
GLM-5 terminal14seed tasks or web pages, Harbor formatrefine agent, manual rubrics; Harbor validationstratified sampling (web route)thousands; build accuracy over 90%
GLM-5 search14web knowledge graph, over 2M pagesbidirectional question-answer verificationdrop if solved by a tool-free model (1 of 8) or an early agentnot stated
Kimi K315knowledge graph; mock applications; AETindependent verifier; per-event criteriahidden verifiers on held-out scenarios (AET)not stated in the sections read
Kimi K2 (SFT data)73,000+ MCP tools; 20,000+ synthetic toolsLLM judge vs per-task rubric; real sandboxes for codenot stated"tens of thousands" of training examples

6.4 What a pass@100 > 0 filter admits

DeepSeek-V3.2 keeps only synthesized tasks with non-zero pass@100 after RL with V3.216. That is a solvability filter, and its arithmetic is worth doing because it is easy to read as a difficulty filter. Let \(p\) be a task's per-sample success probability. The filter passes the task with probability \(1-(1-p)^{100}\), one half at \(p=1-0.5^{1/100}\approx0.0069\). A group of \(G\) samples gives a gradient only if it contains both outcomes (Module 10, Section 5; Module 11), with probability \(1-p^{G}-(1-p)^{G}\). At \(p=0.0069\) and \(G=16\), the group size Nemotron 3 Ultra uses11, that is 10.5% (derived): a task at the edge of the filter yields a usable group about one time in ten, and the filter says nothing about where learning is fastest.

Probability that a task passes a pass@100 filter, and that a group of 8 or 16 samples contains both outcomes, as a function of per-sample success probability on a log axis 00.51 0.0010.010.11 per-sample success probability p (log axis) probability 0.0069 pass@100 at least 1 group of 16 is mixed group of 8 is mixed
Figure 7. Derived curves, not measurements. Blue: probability that a task with per-sample success probability \(p\) passes a pass@100 > 0 filter. Orange and cyan: probability that a group of 16 or 8 samples contains both a success and a failure. At \(p=0.0069\) (dashed line) the filter passes half the tasks and a 16-group is mixed 10.5% of the time. Filter rule from the DeepSeek-V3.2 report16; group size 16 from Nemotron 3 Ultra11.

What a pass@N filter admits, and what the group sees

Task passes pass@N > 0
Group has both outcomes
Group is all-fail

Formulas: \(1-(1-p)^N\) and \(1-p^G-(1-p)^G\), independent samples, fixed per-task \(p\). N = 100 is the report's value; G = 16 is Nemotron 3 Ultra's11.

The arithmetic also shows what the filter catches. A verifier that wrongly rejects correct solutions drives \(p\) to zero and the task is dropped, a useful check. A verifier that wrongly accepts wrong answers raises \(p\) and passes easily, so the filter cannot catch false positives (my reasoning; the report does not discuss it). The check that does is a negative control: run a known-wrong or empty solution and require a reward of 0. None of the formats read here builds that in; R1's code pipeline approximates it by keeping test subsets that expose flaws in known-incorrect submissions (Module 10, Section 3.2).

6.5 The one environment ablation

V3.2's Section 4.3 is the only environment-design ablation in the reports read for this module (K2, K2.5, K3, V3.2, V4, GLM-5, Nemotron 3 Ultra); a search of their texts for "ablat" finds ablations of pre-training, RL infrastructure and distillation, none of environments, harnesses or difficulty filters. For difficulty, it samples 50 general synthesized tasks and evaluates the synthesis model and three frontier models (Table 7). For transfer, it runs RL on the SFT checkpoint using only synthetic general-agent tasks in non-thinking mode, against that SFT model and against V3.2-Exp, trained with RL only on search and code. RL on synthetic data "yields substantial improvements" on τ²-Bench, MCP-Mark and MCP-Universe, whereas "restricting RL to code and search scenarios does not improve performance on these benchmarks"16 (§4.3; the report's Figure 5 plots curves, which I do not quote numerically).

Table 7. Accuracy on 50 randomly sampled synthesized general-agent tasks, pass@K, in %16 (Table 5 of the report).
Pass@KDeepSeek-V3.2-ExpSonnet-4.5Gemini-3.0 ProGPT-5-Thinking
112345162
218476575
426627482

Three limits. The sample is 50 tasks and one run per condition. The comparison arm differs in task content (code and search versus general agent), so the result says synthetic general-agent tasks add something code-and-search RL does not, not that synthetic beats real. And the generalization claim rests on the benchmarks' "environments and toolsets" not having been "encountered during RL training"16 (§4.1), an overlap the reader cannot audit without the environment lists. The task mix is also easy to misread: the report's "over 1,800 distinct environments and 85,000 complex prompts" are different populations. Table 1 of the report counts 24,667 code-agent tasks (real environment), 50,275 search-agent tasks (real environment, synthesized prompts), 4,417 general-agent tasks (synthesized environments) and 5,908 code-interpreter tasks (real), summing to 85,267 (derived). The 1,827 synthesized environments carry 5.2% of the tasks, about 2.4 tasks each (derived)16.

7. What to test before an environment feeds a gradient

The sources contain no end-to-end environment audit, so this section assembles one from checks that individual reports or libraries document, and says where a check has no documented source. Module 19 owns the exploit taxonomy and Module 20 the measurement questions.

7.1 Failures that are not the model's

An agent rollout touches containers, networks, simulated users and judges, and any can fail for reasons unrelated to the policy. GLM-5 is explicit: coding-agent sandboxes "may fail for reasons unrelated to the model (e.g., environment crashes)", and such failures "introduce noisy training signals because they reflect environment instability rather than the model's capability". It records the failure reason per sample, excludes environment failures, and repairs the resulting short GRPO group: "we pad the group by repeating valid samples if the number of valid samples exceeds half of the group size; otherwise, we drop the entire group"14 (§4.1.2). verifiers takes the same stance at the scoring layer: let judge failures propagate rather than score 0, and let a rubric judge return a drop verdict (for example rubric: outcome=task_defect) that "records no reward and drops the trace", so it "counts in no reward mean or training batch"4.

The repair rule has a simple cost curve. If each of \(G\) samples independently fails with probability \(q\), the group is dropped when at most \(G/2\) samples are valid: $$P_{\text{drop}}=\sum_{v=0}^{\lfloor G/2\rfloor}\binom{G}{v}(1-q)^{v}q^{G-v}.$$ With \(G=16\) and \(q=20\%\) (derived), 0.70% of groups are dropped but 96.5% need repair; with \(G=8\) and \(q=30\%\), 19.4% are dropped. Repeated samples carry extra weight in the group statistics, a side effect the report does not discuss (my reading). Independence is also generous: a broken task fails all its samples, so the dropped fraction then equals the fraction of broken tasks, whatever \(G\) is.

Group survival under environment failures

Group dropped
All samples valid
Kept after repair

Rule: keep the group if more than G/2 samples are valid (GLM-514); formula as above. Independent failures are an assumption.

NeMo Gym addresses the same problem from the data side: v0.5.0 added gym eval reverify, which lets you "recompute rewards from stored rollouts without re-running inference", and v0.6.0 added health checks and token, tool-call, turn and latency diagnostics8. Re-scoring stored rollouts makes a verifier fix cheap and answers the versioning problem of Section 4.4.

7.2 Is the task itself valid

Validity checks in the sources fall into four kinds. A reference solution must score full reward: Harbor's oracle run10, verifiers' validate() hook for tasks that have "a known reference solution or gold check that can be verified without calling a model"4, and the gold-patch conditions of SWE-bench, SWE-Gym and V3.2192016. The answer must be unique and checkable: τ-bench writes instructions that admit one outcome3, and GLM-5's search pipeline rejects non-unique answers14. A capable policy must beat chance: TRL advises that "if a capable model can't score higher than a random baseline, the reward signal may need adjustment"2. And the pass-rate profile must sit in a learnable band, which Nemotron 3 Ultra handles by "reward profiling prior to training" and re-profiling when accuracy saturates11 (§3.2, §3.3.2; Module 10 Section 5 covers the filters).

Environment bugs also reach public benchmarks. GLM-5 reports Terminal-Bench 2.0 on a "verified" version that "resolves some ambiguous instructions" alongside the original14; DeepSeek-V4 "acknowledge[s] the environment-related issues noted by GLM-5.1", reports the original dataset for consistency, and gives approximately 72.0 for V4-Pro on the verified subset21 (§5.3.1). If a widely used benchmark needed a verified edition, a training environment built by a synthesis agent and checked by a self-verification loop is unlikely to be cleaner (my inference).

7.3 The checklist, with sources

Table 8. Checks to run on an environment before training on it, each tied to the source that documents it. A row whose source cell says "none" is my addition.
CheckWhat it catchesDocumented in
Reference solution scores full rewardbroken tasks, missing dependenciesHarbor oracle10; verifiers validate()4; V3.2 F2P and P2F16
A wrong or empty solution scores zeroa verifier that is too lenientnone for environments; R1 test filtering (Module 10)
Solvable at least once, not alwaysunsolvable and trivial tasksV3.2 pass@100 > 016; GLM-5 search filters14; K2 pass@k band7
Unique, checkable answerambiguous tasks, multiple valid outcomesτ-bench3; GLM-5 bidirectional check14
A capable model beats randomsign errors, dead rewardsTRL2
Infrastructure failures are excluded, not scored 0noise that looks like model errorGLM-5 rule14; verifiers4
Truncation and timeout have a stated policylength-driven reward noiseverifiers skip flags4; K2 penalty7; TRL total-length cap2
The agent cannot reach the verifiertampering, answer leakageverifiers isolated scoring4; K3 hidden verifiers15
Reward changes are versioned and rollouts re-scorablesilent drift of the objectiveGym versioning rule5; gym eval reverify8
More than one harness per task distributionoverfitting to one tool schema or context ruleK315; Nemotron 3 Ultra11

Where experts disagree

The source map records no dispute about environment design, and this module's evidence explains why: reports describe their environments without ablating them. The one ablation, DeepSeek-V3.2's Section 4.3, compares synthetic general-agent RL against code-and-search RL and nothing else16. What exists is divergent practice with stated reasons and little head-to-head evidence. Four practices are separated below; whether to train under one harness or several has a stated position for diversity (Section 3.2) and none against, so it appears in the assessment.

Binary rewards on final state

TRL's OpenEnv guide (Wordle and Sudoku experiments)2; NeMo Gym's single-step tool verifier8; τ-bench3

A group-relative method uses only ranking inside a group, so partial credit adds a hackable target without adding rank information that a final-state check lacks: "binary rewards (1.0 for success, 0.0 otherwise) gave cleaner training signals than shaped rewards with partial credit"2. NeMo Gym's single-step tool verifier demands exact matches and scores a multi-call response 0, with the stated aim of "eliminating partial-credit reward hacking"8.

  • Few places to hide a hack; easy to audit and re-score
  • Consistent with the shaping theorem: non-potential dense rewards can change the optimal policy17
  • An all-fail group carries no gradient, so hard tasks stall (Module 10)
  • The supporting evidence is a practitioner remark with no numbers2

Graded or shaped rewards where progress is meaningful

Kimi K3's kernel and web tasks15; NeMo Gym's distance, milestone and per-step environments8; Ng et al. on safe shaping17

When the quantity is naturally continuous (speed against a roofline, pixel distance) or full success is rare, a graded reward gives signal that binary does not. K3 gates on correctness first, scales performance second, and adds a hack-detector15. Potential-based shaping is the one form with an optimality guarantee17.

  • Ranks failures, so hard tasks still teach
  • Matches the true objective where it is continuous
  • Partial credit is the classic hack surface: bicycle circles and a vibrating soccer robot17; K3 extended its detector during the run15
  • No ablation in the sources compares the shapes at matched compute

Real, executable environments first

SWE-Gym20; GLM-5's SWE pipeline14; Kimi K2's hybrid design7

Real repositories with real tests give "reliable reward signals" that were "largely absent" in software engineering20. K2 acknowledges "the inherent limitation of simulation fidelity" and adds real sandboxes for coding "where authenticity is crucial"7.

  • Ground truth from tests the repository's authors wrote
  • No simulator to drift from reality
  • Needs a repository history; most agentic domains have none
  • Environment setup cost: 200 human-hours for 2,438 instances in SWE-Gym20; 2026 reports automate it without publishing yields

Synthetic and simulated environments at scale

DeepSeek-V3.216; Kimi K3's mock applications15; Kimi K2's tool simulator7

Where the world cannot be snapshotted, build it: V3.2's synthetic general-agent tasks improved τ²-Bench, MCP-Mark and MCP-Universe where code-and-search RL did not16; K3's mock Gmail, Notion, Slack and Canvas offer "reproducible, large-scale interaction without external APIs or rate limits"15.

  • Breadth independent of repositories or live services
  • Verifier and state under the builder's control
  • Fidelity is unmeasured; self-verification shows consistency, not relevance
  • The one ablation is one run on 50 sampled tasks, against a non-matched arm16
Assessment

Reward shape. Default to a binary reward on final state when a program can read an end state and the pass rate is mid-range. Use a graded reward when the quantity is natively continuous or pass rates are too low for groups to mix, and then do what K3 does: gate on correctness, scale second, and run a hack-detector you expect to extend; make dense per-step credit a potential over state. A matched-compute comparison of binary and partial credit across difficulty bands would settle it; Reasoning Gym has the difficulty knob to run one18, and no source reports it.

Real versus synthetic. The positions are complements: use real artifacts where they exist and synthesize where they do not, with the Section 7 checks on both. V3.2 shows synthetic tasks can transfer to unseen benchmarks, not that they match real tasks per unit of compute. A same-budget comparison of real and synthetic tasks in one domain, evaluated on environments in neither, would settle it.

One harness or several. K3 and Nemotron 3 Ultra argue for diversity1511, and V3.2's context rule shows a harness policy can change what is learned16. No report trains under one harness against several, so the case rests on mechanism (tool schemas and context rules are part of the task), not a measured gap. Training on one harness set and evaluating on a held-out harness would settle it. Until then, Nemotron's hedge is cheap: two or more harnesses per task distribution, at the cost of trace capture for each11.

What is changing

The interfaces are moving faster than the evidence about them. OpenEnv's README says the project "is in early development, so APIs may still change" and lists a technical committee of twelve organizations9. NeMo Gym shipped five minor releases in about four months, dated 06/04, 07/01, 08/06, 09/03 and 10/06/2026 in the README (read as month/day, consistent with the sequence), a mean gap of 31 days (derived)8. What is quoted here about those two, and about the main-branch documentation of verifiers and TRL, will drift within a quarter.

Three trends are visible. Interface convergence at the model boundary: five systems capture tokens there, three of them through a proxy or gateway (Table 3). The unit of environment is widening from a task to task plus harness plus runtime, with harness choice a configuration axis815. And construction is being automated end to end, with setup agents, refine agents and self-verification loops replacing SWE-Gym's 200 human-hours2014, which moves the failure mode from too few environments to too many unaudited ones. Open questions the sources leave: funnel yields for the 2026 pipelines, any ablation of harness diversity, a matched-compute comparison of reward shapes, the fidelity of simulated against real environments, and how reward changes made during a run are versioned.

References

  1. officialFarama Foundation. Basic Usage. Gymnasium documentation, live docs (no version stamp). https://gymnasium.farama.org/introduction/basic_usage/. Accessed 2026-10-27. Used for: Env as an MDP class "not a perfect reconstruction", the reset and step return tuples, terminated versus truncated, and the Space types including Text.
  2. officialHugging Face. OpenEnv Integration for Training LLMs with Environments. TRL documentation, main branch. https://huggingface.co/docs/trl/openenv. Accessed 2026-10-27. Used for: the environment_factory contract, tools versus environments, the reward-design tips (binary versus partial credit, check the final state, test the reward), max_completion_length over a whole conversation, server concurrency, white-box versus harness training.
  3. paperYao, S., Shinn, N., Razavi, P., Narasimhan, K. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045v1, 17 Jun 2024. https://arxiv.org/pdf/2406.12045. Accessed 2026-10-27. Used for: the POMDP formulation and hidden database state (§2), the reward \(r_{\text{action}}\times r_{\text{output}}\) and its stated limitation, the simulated user, Table 1 statistics.
  4. repoPrime Intellect. verifiers documentation: docs/architecture.md, docs/tasksets.md, docs/agent.md (main branch, verifiers.v1 API, no version pinned). architecture.md, tasksets.md, agent.md. Accessed 2026-10-27. Used for: Taskset, Task and TaskData; the rollout lifecycle; shared versus isolated scoring; the interception server; skip-on-truncated and skip-on-timeout; infinite tasksets; reward weights; validate(); the drop verdict.
  5. paperBrockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W. OpenAI Gym. arXiv:1606.01540v1, 5 Jun 2016. https://arxiv.org/pdf/1606.01540. Accessed 2026-10-27. Used for: the 2016 reset and four-element step interface, "environments, not agents", strict versioning of environments (§3).
  6. officialFarama Foundation. Env. Gymnasium API reference, live docs (no version stamp). https://gymnasium.farama.org/api/env/. Accessed 2026-10-27. Used for: the v0.26 split of done into terminated and truncated and its rationale, reset(seed, options), the info field, multi-agent pointer to PettingZoo.
  7. paperKimi Team. Kimi K2: Open Agentic Intelligence. arXiv:2507.20534v2, 3 Feb 2026. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-27. Used for: §3.1.1 agentic data synthesis (SFT), §3.2.1 verifiable-reward gym and sandbox scale, §3.2.3 budget control, §3.3.4 agentic rollout.
  8. repoNVIDIA. NeMo Gym (NVIDIA-NeMo/Gym README). README on main showing release v0.7.0 dated 10/06/2026 (date format unlabeled; read as month/day). https://raw.githubusercontent.com/NVIDIA-NeMo/Gym/main/README.md. Accessed 2026-10-27. Used for: the environment definition, quick-start servers, release notes v0.3.0 to v0.7.0, the environment table (226 rows parsed by me) and the reward shapes in it.
  9. repoMeta-PyTorch with Hugging Face and partners. OpenEnv: Agentic Execution Environments (README). README on main, BSD 3-Clause, "early development". https://raw.githubusercontent.com/meta-pytorch/OpenEnv/main/README.md. Accessed 2026-10-27. Used for: reset, step, state over WebSocket, Docker per environment, white-box versus loop-owning training modes, Harbor TrainingTrace, the maturity statement and the technical committee.
  10. officialHarbor Framework. Create a task (Harbor tutorial). https://docs.harborframework.com/tutorials/create-a-task. Accessed 2026-10-27. Used for: the task directory layout, task.toml timeouts, the reward file, the oracle-agent validation run, RewardKit pointer.
  11. paperNVIDIA. Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv:2606.15007v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-27. Used for: §3.2 unified RLVR, harness diversity, 16 rollouts per sample and reward profiling; §3.3 single-turn rollouts; Appendix A.2 harness robustness.
  12. repoNVIDIA. NVIDIA Nemotron Developer Repository (NVIDIA-NeMo/Nemotron README). README on main, Apache 2.0; Nemotron 3 Ultra section. https://raw.githubusercontent.com/NVIDIA-NeMo/Nemotron/main/README.md. Accessed 2026-10-27. Used for: the Ultra recipe scope (Pretrain → SFT), the "Open-Source Data Only" notice and its statement that results differ from the tech report benchmarks.
  13. paperKimi Team. Kimi K2.5: Visual Agentic Intelligence. arXiv:2602.02276v2. https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-27. Used for: Appendix D, the Gym-like interface, white-box and black-box environments, LLM Gateway, token-in-token-out.
  14. paperGLM-5 Team (Zhipu AI and Tsinghua University). GLM-5: from Vibe Coding to Agentic Engineering. arXiv:2602.15763v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-27. Used for: §4.1.2 TITO gateway and the environment-failure repair rule, §4.2.1-§4.2.3 SWE, terminal and search environment pipelines, the verified Terminal-Bench 2.0 note.
  15. paperKimi Team. Kimi K3: Open Frontier Intelligence. arXiv:2607.24653v2, 7 Aug 2026. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-27. Used for: §4.2.1 white-box harness, §4.2.2 knowledge-graph synthesis, §4.2.4 kernel tasks, §4.2.5 mock applications, §4.2.6 Autonomous Execution Tasks, §4.2.7 web development.
  16. paperDeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-27. Used for: §3.2.1 context management, §3.2.3 Table 1 and the code and general-agent environment pipelines, §4.1 generalization claim, §4.3 Table 5 and Figure 5.
  17. paperNg, A. Y., Harada, D., Russell, S. Policy invariance under reward transformations: Theory and application to reward shaping. ICML 1999. https://people.eecs.berkeley.edu/~pabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf. Accessed 2026-10-27. Used for: Theorem 1 (§3) on potential-based shaping and the bicycle and soccer examples (§1); read from the PDF pages as images.
  18. paperStojanovski, Z., Stanley, O., Sharratt, J., Jones, R., Adefioye, A., Kaddour, J., Köpf, A. Reasoning Gym: Reasoning Environments for Reinforcement Learning with Verifiable Rewards. NeurIPS 2025 Datasets and Benchmarks; arXiv:2505.24760v2, 20 Oct 2025. https://arxiv.org/pdf/2505.24760v2. Accessed 2026-10-27. Used for: design principles P1-P3 (§2), zero-shot scores and the difficulty cliff (§3, Figure 3), curriculum experiment and Table 5 (§5), limitations (§7).
  19. paperJimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024; arXiv:2310.06770. https://arxiv.org/pdf/2310.06770. Accessed 2026-10-27. Used for: the three-stage construction pipeline and resolution criterion (§2), Table 10 (93,139, 11,407, 2,294), execution-based validation (Appendix A.3).
  20. paperPan, J., Wang, X., Neubig, G., Jaitly, N., Ji, H., Suhr, A., Zhang, Y. Training Software Engineering Agents and Verifiers with SWE-Gym. arXiv:2412.21139v2, 6 Jun 2025. https://arxiv.org/pdf/2412.21139v2. Accessed 2026-10-27. Used for: §3 and §3.1, the 2,438 validated instances, 64,689 raw instances, 200 human annotation hours, 10,000 CPU core hours, 6 TB of images.
  21. paperDeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348v1. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-27. Used for: the Terminal-Bench 2.0 environment-issue note and verified-subset score (§5.3.1).