Reinforcement Learning for LLM Post-Training · Module 17
RL Environments for LLMs: Task Datasets, Harnesses, Verifiers, State, and Reward Shaping
An RL environment is the part of the training loop that is not the model: the tasks, the program through which the model acts, the state those actions change, and the verifier that turns the final state into a number. This module rebuilds the Gymnasium contract for token and tool-call actions and reads how NeMo Gym, verifiers, OpenEnv, Reasoning Gym and seven 2025-2026 technical reports construct, shape and validate environments. Afterwards you can specify an environment for a new domain (tasks, harness, verifier, state, reward) and name what to test before it feeds a gradient.
environment_factory (the trainer drives generation and tool calls; the reward is read from the environment after the episode)2, τ-bench's hidden database state and LLM-simulated user3, and the verifiers library's separation of harness, runtime and scoring4. The orange arrows have no Gymnasium counterpart: scoring happens once, from state the agent never sees directly.1. From reset and step to a trajectory of turns
1.1 What the Gymnasium contract promises
The 2016 OpenAI Gym whitepaper fixed the shape every environment library since has copied: env.reset() returns the first observation, env.step(a) returns (observation, reward, done, info), and the library abstracts the environment only, not the agent. It also set a rule that matters for LLM work: any change to an environment's behavior must come with a new version name, because results before and after are incomparable5. Gymnasium split done into two flags in v0.26 "to make it clearer to users when the environment had terminated or truncated", which it calls "critical for reinforcement learning bootstrapping algorithms". terminated means the agent reached a terminal state of the task's MDP, positive or negative; truncated means a condition outside the MDP held, typically a time limit; either requires a reset()6. reset returns (observation, info) and takes seed and options; info may hold "individual reward terms that are combined to produce the total reward"6. The docs call Env an MDP "not a perfect reconstruction, missing several components"1.
The contract carries four assumptions that were free in Atari and MuJoCo and are not free for an LLM: the environment owns dynamics and reward; the action space is declared up front as a Space (Box, Discrete, Dict, and a Text space for strings with minimum and maximum length1); one step takes one action and returns one reward; and stepping is cheap. An agent that emits thousands of tokens per action on a machine that must be built, snapshotted and graded violates the last two (my reading; the docs make no such claim).
1.2 What the LLM does to each element
Module 2 gave two ways to cast a single completion into the contract: a token-level MDP and a bandit whose single action is the whole completion. A tool-using agent adds a third unit, the turn. A trajectory is \(\tau=(x,a_1,o_1,a_2,o_2,\dots,a_K)\): each \(a_k\) is a token sequence sampled from \(\pi_\theta\) (reasoning plus a tool call or final answer) and each \(o_k\) is text the environment wrote. The log-probability splits into a policy part and an environment part, $$\log p_\theta(\tau)=\sum_{k}\log\pi_\theta(a_k\mid x,a_{\lt k},o_{\lt k})+\sum_{k}\log P(o_k\mid x,a_{\le k},o_{\lt k}),$$ and only the first sum depends on \(\theta\). So a unit test, a browser or a simulated user can sit in the loop without being differentiable, as Module 2 argued for a single completion. Not every token in \(\tau\) is a policy token, though: the \(o_k\) segments sit in the context but were not sampled, which is where loss masks come from (Module 18).
| Element | Single completion | Multi-turn tool use |
|---|---|---|
| observation | prompt, then tokens so far | user message, tool output, page text; history re-enters the context |
| action | next token, or whole completion | one model turn: reasoning plus a parsed tool call or answer |
| action space | the vocabulary | the tool schema: names and typed arguments |
reset | sample a prompt | load a task, build or restore the sandbox, return the first observation |
step | append a token | execute the call in the sandbox, return its output as text |
| reward | reward-model score at EOS, or verdict | verifier on final state; sometimes per-step terms |
terminated | EOS | model submits; task solved or failed |
truncated | max_new_tokens | context cap, turn cap, wall-clock timeout |
| hidden state | none | files, database rows, processes, browser session |
Two rows deserve a closer look. The action space is now a schema, not a vocabulary. TRL discovers every public method of the environment class, exposes it as a function-calling tool, requires typed docstrings to build the schema, and advises against a generic step(action) because "the model needs meaningful tool names and argument descriptions to learn tool calling"2. And the state is no longer the observation. τ-bench writes its environment as a POMDP with \(S=S_{db}\otimes S_{user}\), \(A=A_{db}\cup A_{user}\), \(O=O_{db}\cup O_{user}\) and reward \(R:S\to[0,1]\); the database is "hidden from the agent and the user" and reachable only through API actions, while the user is a language model (gpt-4-0613) whose state is its prompt plus the conversation3. An environment can contain a model. The context window is a log of observations standing in for a belief state; the true state is in the sandbox, where the verifier reads it.
1.3 Terminated, truncated, and what to do with a cut-off episode
Gymnasium's split exists so that a bootstrapping learner can treat a time-limit cut differently from a real end6 (Module 2 explains why for a critic). A critic-free recipe faces a different question: should a truncated trajectory be graded at all? The sources document three answers. Grade it as usual, the default when the verifier reads whatever state exists. Skip it: verifiers has skip_on_truncated and skip_on_timeout switches that leave a cut-off run with no rewards, so its reward is 0 and trace.info["scoring_skipped"] records why4. Or penalize it: Kimi K2 enforces a per-sample token budget by task type, and "responses that exceed this token budget are truncated and assigned a penalty"7 (§3.2.3). TRL adds a trap for multi-turn work: max_completion_length caps tokens across the whole conversation, "all model generations + tool results combined", so verbose tool output can exhaust it2. No source states a consensus on which flag to set or what reward should follow.
class SqlEnv:
"""One instance per rollout. State lives in the sandbox, not in the context."""
def reset(self, task):
self.db = sandbox.restore(task["snapshot"]) # task = one dataset row
self.gold, self.turns = task["gold_rows"], 0
return task["question"], {"task_id": task["id"]} # (observation, info)
def step(self, action): # action = one parsed model turn
self.turns += 1
if action.tool == "run_sql": # exploration: no reward yet
obs = self.db.execute(action.args["query"]) # tool output returns as text
return str(obs), 0.0, False, self.turns >= 8, {}
if action.tool == "submit": # final answer, graded from state
rows = self.db.execute(action.args["query"])
r = float(sorted(rows) == sorted(self.gold)) # unordered result-set equality
return "", r, True, False, {"verifier": "result_set_equality"}
return "unknown tool", 0.0, False, self.turns >= 8, {}
The contract as a sketch (mine, not any library's API). The reward rule mirrors the Bird SQL verifier in NeMo Gym's catalogue (Section 4.2). Everything hard is hidden in sandbox.restore, execute and the dataset; the rest of this module is about those three.
2. Anatomy: tasks, harness, state, verifier
2.1 Four parts under four vocabularies
NVIDIA's NeMo Gym defines an environment as "the complete system an agent interacts with to complete a task": a dataset (the tasks), an agent harness ("how the model interacts with the world"), a verifier and per-task state. Its quick start starts three servers, a mcqa resources server, a mcqa_simple_agent agent and a policy_model, so, as far as the README shows, tasks and verifier live in the resources server, the harness in the agent server and the model in a third (my reading of the layout). The README is candid about overkill: "if you're scoring model outputs with a stateless check and don't need scale or training, a script is probably sufficient"8.
The verifiers library cuts the same object differently. A Taskset loads Tasks from a dataset or generator; a Task holds the lifecycle hooks (setup, finalize), tools, stop conditions, metrics and rewards; TaskData is the immutable record of prompt, reference answer, artifacts and per-task limits; the agent solves it "inside a harness running in a runtime" (subprocess, Docker, remote sandbox or custom)4. OpenEnv cuts at the network boundary: each environment is a Docker image exposing reset(), step() and state(), sync or async, over a WebSocket9. Table 2 lines them up; the rows for NeMo Gym, verifiers and OpenEnv are snapshots of main-branch documents and will drift.
| Library | Unit | Interface | Isolation | Note |
|---|---|---|---|---|
| Gymnasium6 | Env class | reset, five-element step | in-process | single-agent |
| OpenEnv9 | Docker image plus client | reset, step, state over WebSocket; MCP at /mcp | container per environment | "early development"; 40+ environments |
| NeMo Gym8 | dataset + harness + verifier + state | resources, agent and model servers | seven sandbox providers by v0.5.0 | README shows v0.7.0 (10/06/2026) |
| verifiers4 | Taskset of Tasks in a runtime | async Python; @vf.reward(weight), @vf.metric, built-in judges | subprocess, Docker, remote; optional isolated scoring | docs on main, no version |
| Harbor10 | task directory | harbor run with an "oracle" agent | Docker, Daytona, Modal | tutorial shows rewards 0 and 1 |
TRL environment_factory2 | Python class per generation | reset(**row); public methods become tools | none by itself | OpenEnv servers default to 1 session |
2.2 One rollout as a lifecycle
Whatever the vocabulary, a rollout passes through the same stages, and the order is where environments go wrong. The verifiers documentation states the lifecycle most explicitly, and the stepper below follows it.
- Load the task. The taskset yields tasks from a dataset or generator; the client ships each
TaskDatato the worker that runs it. A generator can declare itself infinite, and every run must then bound it4. - Set up. Task
setupprepares files, repositories or services in the sandbox, and the harness is installed in the same runtime4. Harbor builds the image fromenvironment/Dockerfile, allowing 600 s in its tutorial example10. - Run the agent loop. The model and tools run until completion or a stop condition. verifiers routes model traffic through an interception server that builds traces live, which "prevents agents tampering with their own traces", and can run stop hooks that rewrite tool responses "for example, to block reward hacks"4.
- Finalize.
finalize"captures any outputs or state needed for scoring and inspection before tests mutate the workspace"; for software tasks it saves the agent's git diff first. A run stopped mid-turn has its harness closed before grading, so the agent cannot edit during it4. - Score. Metrics run first and affect nothing; rewards sum as score x weight (weight 0 logs a signal without training on it); judges run last. If a judge fails, the documentation says to raise "instead of returning 0.0, which would record an infrastructure failure as a wrong answer by the model"4.
- Emit the trace. The trainer receives messages, rewards, metrics and exact token ids captured at the model call, not re-tokenized from text. NeMo Gym v0.6.0 describes preserving "exact token IDs across multi-step runs" with external harnesses, and v0.7.0 adds durable worker-side capture for vLLM and Megatron8. What the loss does with them is Module 18.
2.3 A concrete spec: the Harbor task directory
The cleanest statement of what a terminal-style task must contain is Harbor's tutorial, the format GLM-5's terminal environments use (Section 6.3). A task is a directory:
ssh-key-pair/
|-- instruction.md what the agent receives
|-- task.toml metadata; [verifier] timeout_sec = 120.0, [agent] timeout_sec = 120.0,
| [environment] build_timeout_sec = 600.0
|-- environment/Dockerfile the container the agent works in
|-- solution/solve.sh reference solution, run by the "oracle" agent
`-- tests/
|-- test.sh verifier entry point; must write /logs/verifier/reward.txt
`-- test_outputs.py pytest checks run by test.sh
The tutorial's test.sh writes 1 to the reward file if pytest exits 0 and 0 otherwise; for "multi-criterion verifiers, weighted scoring, or LLM-as-a-judge rubrics" it points to a separate library, RewardKit10. The validity check is built in: harbor run -p ssh-key-pair -a oracle executes solve.sh and a reward of 1 is the pass condition10. A task whose reference solution does not score 1 is broken whatever a policy later does with it; every pipeline in Section 6 contains this check under another name.
2.4 A census of one catalogue
The NeMo Gym README's environment table is the largest public list of RL environments among the sources. As of the README fetched on 2026-10-27 it has 226 rows under 152 distinct names; repeated names are variants (Reasoning Gym appears nine times, the SWE-bench agent seven, single-step tool use seven) (my count; Figure 3). Agentic, coding and knowledge rows dominate, and 202 of the 226 rows show "-" in the Dataset column, which the README defines as data "not publicly released yet" or "procedurally generated using a provided script"8. The environment code is public and most of the environment data is not (derived: 24 rows link a public dataset). The README's "battle-tested in production Nemotron training" is a vendor claim; Nemotron 3 Ultra's report says it open-sources recipes, data and RL environments with the checkpoints11. The Nemotron repository README (fetched 2026-10-27) is narrower: its Ultra recipe "covers Pretrain → SFT", trains "exclusively on the open-sourced subset of training data", and states "Results will differ from the tech report benchmarks, which used additional proprietary data"12. RL environments may sit in the NeMo Gym or NeMo RL repositories; I inspected neither beyond the NeMo Gym README (Module 32).
3. The harness: who owns the loop
3.1 White-box loops and loop-owning harnesses
The harness is the program between the model and the sandbox: it renders the prompt, parses tool calls, runs them and decides what stays in context. OpenEnv's README names two ways to put one in a training run. In "environments as tools (white-box)" the trainer runs the multi-turn loop and the environment supplies tools and reward. In "real agent harnesses (loop-owning)" the agent runs its own loop and the integration "captures every model call as a framework-neutral TrainingTrace"9. TRL's guide draws the same line, calling its environment_factory path white-box and keeping rollout_func for external agent servers that own the loop2. Kimi K2.5 has built-in white-box environments and black-box ones "that can only run under standard LLM API protocol"13 (Appendix D).
The black-box case needs more than text: the trainer needs the exact token ids the model sampled (Module 18 covers why). Five systems solve it at the model boundary rather than inside the harness (Table 3); three of them describe the device as a proxy or gateway that speaks the harness's native API and records tokens in passing. verifiers' interception server forwards "using the API surface that the harness expects, such as OpenAI Responses for Codex or Anthropic Messages for Claude Code"4; NeMo Gym lists Claude Code and Codex against a model server's /v1/messages and /v1/responses8; K2.5's LLM Gateway "keeps detailed records of rollout requests and responses"13; GLM-5's TITO gateway "intercepts all generation requests from rollout tasks and records each trajectory's token IDs and metadata"14 (§4.1.2). System Design Modules 27 and 37 cover the harness, sandbox and serving engineering.
| System | Trainer-run loop | Harness-run loop | Where tokens are recorded |
|---|---|---|---|
| OpenEnv + TRL92 | environment_factory | Harbor integration: Claude Code, Codex, OpenCode | TrainingTrace per model call |
| verifiers4 | harness in the runtime | any harness with a provider API | interception server, live traces |
| NeMo Gym8 | in-tree agents (OpenHands, LangGraph) | Claude Code, Codex via /v1/messages, /v1/responses | exact token ids (v0.6.0); worker-side capture (v0.7.0) |
| Kimi K2.513 | built-in white-box environments | black-box via LLM Gateway | gateway logs; token-in-token-out |
| GLM-514 | rollout logic in the framework | external agent frameworks | TITO gateway, per trajectory |
3.2 Harness diversity as a training-data property
Two reports treat the harness as something to vary. Kimi K3: "training with a single fixed agent harness can cause a model to overfit to a particular tool schema, system prompt, context management mechanism, or interaction protocol". Its white-box environment represents a harness as "a collection of configurable, composable modules" (tools, system prompts, context strategies, skills, memories, subagents), instantiates Kimi Code, Claude Code, Codex, OpenClaw and Hermes "as well as entirely new ones", and varies configurations by task group15 (§4.2.1); web-development tasks use "diverse agent scaffolds rather than a single fixed harness"15 (§4.2.7). Nemotron 3 Ultra sorts its tasks into five verticals and trains each under at least two of six harness families (Stirrup, OpenHands, OpenCode, Terminus, Droid, internal), "reducing overfitting to any particular harness design"11 (§3.2, Appendix A.2). NeMo Gym shows the mechanism in configuration: the same Reasoning Gym tasks pair with Claude Code, Codex, and LangGraph orchestrator, parallel-thinking, reflection and ReWOO agents8. With the harness a config axis, diversity costs a YAML file instead of a new environment.
The harness changes the task itself. DeepSeek-V3.2 keeps historical reasoning when only tool outputs are appended and discards it when a new user message arrives; frameworks that simulate tool results as user messages, such as Roo Code or Terminus, "may not fully benefit", so the report recommends non-thinking mode for them16 (§3.2.1). One model under two context policies solves two different sequential problems, which argues for diversity and for pinning harness versions in reports. Neither K3 nor Nemotron states a harness version in the sections read.
3.3 Environments that are not interactive
Not every agent environment runs a sandbox end to end. NeMo Gym's "PivotRL" verifiers score one proposed tool call against a demonstrated action: a single-step search verifier requires queries to match a golden word bag exactly, applies list-F1 guardrails, and scores a response with more than one tool call as 0, aiming at "eliminating partial-credit reward hacking"8. Nemotron 3 Ultra reports "single-turn rollouts, similar to PivotRL" for most agentic tasks in its distillation stage11 (§3.3; Module 18 gives its reason). The trade is in the catalogue's own phrase, "behavior cloning per step": no sandbox cost or long-horizon credit assignment, but the target becomes matching a demonstration8.
4. The verifier reads state, and the reward has a shape
Module 10 covers how to build a verifier for an answer string, a test suite or a constraint, and Module 16 covers judges. This section covers what is specific to an environment that has state: what the verifier is allowed to read, which reward shapes the shipped environments use, what shaping theory says about dense rewards, and the gates reports build into verifiers.
4.1 Grade the state, not the transcript
τ-bench is the cleanest worked example of a state-based reward. Each task has a hidden annotation of the ground-truth database writes; the simulated user's instruction is written so "only one possible outcome" exists under the domain policy; and the reward is a product of two binary checks, \(r=r_{\text{action}}\times r_{\text{output}}\in\{0,1\}\): the final database must equal the unique ground-truth database, and the agent's replies must contain the required strings. Dialogue and read-only calls can vary freely3. The paper's example succeeds if the only write is one specific return_delivered_order_items call and the messages contain "54.04" and "41.64". The benchmark has 115 retail tasks (500 users, 50 products, 1,000 orders) and 50 airline tasks (500 users, 300 flights, 2,000 reservations)3 (Table 1). The authors flag the residual false positive: \(r=1\) "might be a necessary but not sufficient condition", since the agent could issue the return without explicit user confirmation3. Reading state removes transcript ambiguity and leaves the policy-compliance gap.
Kimi K3's Autonomous Execution Tasks state the principle in one line: "rewards are grounded in the verifier's evaluation of the final environment state rather than the agent's self-reported completion"15 (§4.2.6). TRL's reward-design tips say the same in practice: "check the final state, not the path", so the model is free to discover its own strategies2.
4.2 A catalogue of reward shapes
Module 10 (Section 3.4) reads NeMo Gym's Bird SQL, Spider2, Blackjack, GUI-coordinate and single-step tool-use verifiers. Table 4 adds the other shapes that appear in the shipped environments and the reports, and Figure 4 draws the shape classes. The plots are schematics: where a source states only the shape in words, I draw that shape and say so.
| Environment | Reward as described | Shape |
|---|---|---|
| Blackjack8 | +1 win, 0 draw, -1 loss or bust | sparse ternary |
| Bird SQL, Spider2 Lite8 | binary, from unordered result-set equality; no LLM judge | binary on state |
| GUI coordinate8 | "smooth quadratic reward based on Euclidean distance" | smooth, distance-based |
| ToolSandbox8 | milestone similarity in [0, 1]; 0 if a minefield is hit | graded with a veto |
| OpenAir congestion8 | dense per-step KPI reward; the agent sums step rewards into the episode return | dense |
| Long-form MT (LongMT)8 | reference-free COMETKiwi score of a document-level translation | learned metric as verifier |
| Abstention8 | three-tier reward with an LLM judge; rewards abstaining over answering wrongly (Module 24) | tiered |
| τ-bench3 | \(r_{\text{action}}\times r_{\text{output}}\in\{0,1\}\): database equals ground truth and replies contain required strings | product of binary state checks |
| K3 kernel tasks15 | 0 above a numerical-error threshold; 0.5 at expert performance, rising toward 1 near the hardware roofline; hack penalties | gated, then performance-scaled |
4.3 What shaping theory says about dense rewards
Ng, Harada and Russell's 1999 result is why dense rewards draw suspicion. Their Theorem 1 says a shaping reward \(F\) preserves optimal policies, without prior knowledge of the MDP, if and only if it is potential-based, \(F(s,a,s')=\gamma\Phi(s')-\Phi(s)\)17 (§3). The paper opens with two bugs from non-potential shaping: a bicycle agent rewarded for moving toward the goal learned to ride in tiny circles, and a soccer robot rewarded for touching the ball learned to stay beside it and "vibrate"17 (§1).
With \(\gamma=1\), the usual setting for LLM episodes (Module 2), a derivation of mine shows what a potential does: the shaping terms telescope, $$\sum_{t=0}^{T-1}F(s_t,a_t,s_{t+1})=\Phi(s_T)-\Phi(s_0).$$ A potential-based dense reward is therefore a function of the final state (up to the constant \(\Phi(s_0)\) for a fixed start) added to the sequence return, and for a group-relative method only the differences of \(\Phi(s_T)\) across the group survive. A safe dense reward carries no more path information than a terminal reward on a final-state score \(\Phi\), so the design question is the choice of \(\Phi\): partial credit by another name. A milestone count credited whenever it is reached is not a potential and can be farmed by revisiting, the cycle argument of the 1999 paper.
Partial credit on the final output is a different object: another terminal reward, whose maximizer matches the original when the top score is reachable only by a correct output. It changes the ordering among failures, which binary treats as equal. TRL reports that in Wordle and Sudoku "binary rewards (1.0 for success, 0.0 otherwise) gave cleaner training signals than shaped rewards with partial credit", arguing that "GRPO compares completions within a group, so the relative ranking matters more than the absolute values", but gives no numbers2. Module 10 (Section 4) works the group arithmetic.
4.4 Gates and tamper resistance built into the verifier
Module 19 owns the exploit taxonomy; this lists the designs. K3's kernel tasks zero the reward above a numerical-error threshold, score performance against an expert implementation, and run "a hacking-detection system that penalizes reward-hacking strategies such as CUDA graph replay, input caching, and precision reduction", extended "as new hacking strategies are observed"15 (§4.2.4). The reward function therefore changed during the run; Gym's versioning rule (Section 1.1) would call that a new environment, and the report does not say how versions were tracked (my reading). K3's Autonomous Execution Tasks isolate agents from verifiers, pair "public verifiers that offer diagnostic feedback with hidden verifiers that evaluate held-out scenarios", and use "penalty-based rewards under limited submission budgets"15 (§4.2.6). verifiers offers isolated scoring: artifacts restored into a fresh container, retried up to 2 times by default on retryable errors4. GLM-5's terminal pipeline has a refine agent check that "the environments are robust against potential exploits or shortcuts"14 (§4.2.2).
5. Procedural environments: a generator, a verifier and a difficulty knob
5.1 An environment as a distribution, not a list
Reasoning Gym (Stojanovski et al., NeurIPS 2025 Datasets and Benchmarks) replaces the fixed dataset with a program: "over 100 data generators and verifiers" across algebra, arithmetic, computation, cognition, geometry, graph theory, logic and games, with "the ability to generate virtually infinite training data with adjustable complexity"18. Its principles: every task algorithmically verifiable (P1); large solution spaces, "rewarding generalizable strategies above overfitting and mitigating reward hacking" (P2); and "configurable parameters systematically control problem characteristics, enabling dynamic curricula via precise difficulty adjustment" (P3)18 (§2). Parameters come as difficulty (node counts, polynomial degrees, word lengths), structural (dimensionality, constraint types, proof depth) and stylistic (variable names, number formats, framing), the last varying presentation without changing difficulty18. verifiers has the same pattern as a taskset whose load() is a generator with INFINITE = True, which forces every run to bound it4; NeMo Gym wraps Reasoning Gym as a resources server8.
The environment is thus a distribution \(x\sim\mathcal D(\kappa)\) indexed by a configuration \(\kappa\) that the loop can move while it runs. Module 10 (Section 5) says why that matters: a prompt yields a gradient only when a group mixes successes and failures, so the useful band of \(\kappa\) is where the pass rate is neither near 0 nor near 1.
5.2 The cliff and the curriculum
Two measurements in the paper locate that band. On hard configurations the zero-shot ranking separates reasoning models (o3-mini 63.5%, DeepSeek-R1 59.5%, Grok 3 Mini 55.1%) from general-purpose ones (Llama 4 Maverick 41.5%, Claude 3.5 Sonnet 40.3%, Gemma 3 27B 20.3%)18 (§3). And moving from easy to hard parameter settings produces what the authors call a difficulty cliff: DeepSeek-R1's score falls by 61.8 on code, 29.6 on graphs, 27.9 on algorithms and 11.8 on geometry; o3-mini's by 71.9, 33.8, 25.6 and 33.118 (the paper's Figure 3b, reported as "Δ score (%)").
The curriculum experiment trains Qwen2.5-3B-Instruct with GRPO under two regimes. The curriculum starts at the easiest level and advances "when performance exceeds 70% over 20 training steps"; the baseline samples uniformly from all levels; both train for one epoch and are evaluated on 50 held-out examples per level18 (§5). Figure 5 plots two of the three environments in the paper's Table 5; Table 5 below reproduces all three.
| Environment (level = …) | Level | Base | Uniform | Curriculum | Gain (pp) |
|---|---|---|---|---|---|
| Spell Backwards (word length) | 4 | 12.00 | 30.00 | 70.67 | +40.67 |
| 6 | 3.33 | 10.67 | 30.00 | +19.33 | |
| 8 | 0.00 | 1.13 | 3.37 | +2.24 | |
| 10 | 0.00 | 0.01 | 0.01 | +0.00 | |
| Mini Sudoku (empty cells) | 4-6 | 1.13 | 54.00 | 56.00 | +2.00 |
| 6-8 | 0.00 | 25.33 | 28.00 | +2.67 | |
| 8-10 | 0.00 | 6.67 | 20.00 | +13.33 | |
| 10-12 | 0.00 | 1.13 | 5.33 | +4.20 | |
| Count Primes (number range) | 100-500 | 12.00 | 4.00 | 30.67 | +26.67 |
| 100-1000 | 3.33 | 5.03 | 12.67 | +7.64 | |
| 100-5000 | 1.33 | 0.00 | 3.03 | +3.03 |
Three things follow. The gain is largest where the base model is mid-range and vanishes at the cliff: Spell Backwards at length 10 is 0.01% under both regimes, so one epoch of curriculum does not cross it. The curves show the authors' mechanism: "increases in difficulty level are followed by a sharp drop in reward", and in Count Primes the curriculum model "never advanced past the initial level" yet beat uniform sampling at every level18 (the paper's §5 and its Figure 6). My reading, consistent with Module 10's zero-variance argument and not stated by the paper, is that uniform sampling spends part of each batch where the pass rate is near zero and groups carry no gradient. The authors also caution that a curriculum's value "may be more limited in environments where progression paths are ambiguous or difficult to formalize"18: a difficulty knob is a property of the domain, and most agentic domains lack one.
Other results bound the pattern. RLVR on one category transfers within it, and an RG-Math model trained on Qwen2.5-3B-Instruct gains 9.7 on MATH, 7.7 on Big-Bench Hard and 0.5 on GSM8K (the paper's "%"), at "approximately 1500 A6000 hours" for all experiments18. The authors list limits: knowledge- or creativity-heavy domains resist generators; verification functions "may not capture all aspects of solution quality"; and the library "focuses on single-turn, text-based reasoning"18 (§7). Module 20 covers how to read transfer claims.
6. Building environments at scale
Environment construction is the least documented part of the stack in the comparative sense: every report describes its pipeline, almost none ablates it, and the numbers that exist are scale counts rather than quality measurements. Table 6 collects each pipeline's source, validity gate and difficulty filter; the text reads them in order of how much each discloses.
6.1 The funnel, priced
SWE-bench fixed the template for turning repositories into verifiable tasks. It keeps merged pull requests that resolve an issue and edit test files, runs the PR's tests before and after its other changes, and keeps an instance only if at least one test flips from fail to pass and installation succeeds; it drops tasks whose tests call names first introduced by the solution, since resolving those "may be an impossible task even for human developers"19 (§2.1, Appendix A.3). Its Table 10 funnel runs from 93,139 crawled pull requests through 11,407 task instances to 2,294 validated ones: yields of 2.46% and 20.1% (derived). A patch resolves an issue only if it applies and all the task's tests pass19 (§2.2).
SWE-Gym, the first training-oriented version, says why the validated fraction is the scarce part: its 64,689 raw instances over 358 repositories "do not contain executable environments or a guarantee that [their] unit tests are effective", so the authors "semi-manually create executable environments" for 11 repositories20 (§3.1). Validation (the gold patch must pass more unit tests than the original code) "required approximately 200 human annotation hours and 10,000 CPU core hours" for 2,438 instances shipped as 6 TB of Docker images: about 4.1 CPU core-hours, 4.9 human-minutes and, if TB means 1012 bytes, 2.5 GB per instance (derived). Its repositories are disjoint from SWE-bench's to avoid contamination20.
6.2 Replacing the human hours with a setup agent
The 2025-2026 reports automate that step. DeepSeek-V3.2 mines "millions" of issue-PR pairs, filters them by heuristics and LLM judgments, and has a V3.2-powered setup agent install packages, resolve dependencies and run tests, with JUnit output so parsing is uniform. The gate is stated precisely: "an environment is deemed successfully built only when applying the gold patch results in a non-zero count of false-to-positive (F2P) test cases … and a zero count of pass-to-fail (P2F) test cases (indicating no regressions)"; the result is "tens of thousands" of environments across languages that include Python, Java, JavaScript, TypeScript, C, C++, Go and PHP16 (§3.2.3). GLM-5 uses the RepoLaunch framework and LLM-written log parsers to extract fail-to-pass and pass-to-pass tests, and reports over 10,000 environments across thousands of repositories and nine languages14 (§4.2.1). Neither gives a funnel yield, so SWE-bench's 2.46% cannot be compared with the 2026 pipelines.
6.3 Environments with no repository to start from
Most agentic domains have no pull-request history. Four reports show four ways to manufacture the tuple of environment, task and verifier.
DeepSeek-V3.2, general agents. A synthesis agent gets a task category and a sandbox with bash and search, fetches data into a sandbox database, writes task-specific tools as functions, then proposes a simple task with a Python solution function and a Python verification function. The solution may call only the tools or do logic, "and cannot call other functions or directly access the database, ensuring the task can only be solved through the tool interface"; its output must pass the verifier, and the agent edits either until it does. It then raises the difficulty, augmenting the toolset when needed (Figure 6). The target is tasks that are "hard to solve but easy to verify", illustrated by a trip-planning task with a 15-function tool set and a constraint check on the submitted itinerary16 (§3.2.3).
GLM-5, terminal and search. Terminal tasks come from two pipelines. From seed tasks, an LLM drafts verifiable tasks, a construction agent writes them in Harbor format, and a refine agent iterates against manual rubrics so images build, tests match the task text and shortcuts fail: "thousands" of environments with "Docker construction accuracy exceeding 90%". From web pages, a coding agent writes a task and runs the Harbor validation script on its own output until it passes; "only tasks that successfully clear this self-verification loop are admitted"14 (§4.2.2). Such a loop shows a task is internally consistent, not that it is worth learning from (my reading). Search questions come from a web knowledge graph over more than two million pages, then a three-stage filter: drop a question if a tool-free reasoning model answers it correctly in at least one of eight attempts; drop it if an early-stage agent solves it in a few steps; have a verification agent validate question and answer "bidirectionally", rejecting non-unique answers14 (§4.2.3).
Kimi K3. Agents grow a hierarchical knowledge graph by web search; sampled nodes drive retrieval of public materials, and a synthesis agent writes tasks of a chosen type. Personal-assistant tasks run in mock Gmail, Notion, Slack and Canvas that "preserve the core semantics of their real-world counterparts", over multiple simulated days with dozens of interdependent events, each with its own rule or LLM criterion; "a single rollout may involve up to thousands of tool calls and millions of context tokens". Autonomous Execution Tasks specify an initial state, a constrained goal, a tool-based action space, budgets and an independent verifier, and show the agent no "reference trajectories or predefined procedures"15 (§4.2.2, §4.2.5, §4.2.6).
Kimi K2: a correction to the source map. K2's pipeline (3,000+ real MCP tools, 20,000+ synthetic tools evolved from domain hierarchies, thousands of agents, a rubric per task, LLM-simulated users, and a tool simulator "functionally equivalent to a world model" that keeps state) sits in the report's supervised fine-tuning section. It produces demonstrations: an LLM judge keeps only trajectories that meet the rubric, "effectively implement[ing] large-scale rejection sampling", with real sandboxes where "authenticity is crucial"7 (§3.1.1). The RL section describes something else: a Gym-like framework, many verifiable tasks, a Kubernetes sandbox service with "over 10,000 concurrent sandbox instances", and a self-critic reward7 (§3.2). The report does not say the simulated environments are reused for RL, and this module does not assume it.
| Pipeline | Source material | Validity gate | Difficulty filter | Reported scale |
|---|---|---|---|---|
| SWE-bench19 | PRs from 12 Python repositories | at least one fail-to-pass test; install and runtime succeed | none (benchmark) | 93,139 to 11,407 to 2,294 |
| SWE-Gym20 | PRs from 11 Python repositories | gold patch passes more tests; semi-manual setup | not stated | 2,438 validated; 64,689 raw |
| V3.2 code agent16 | millions of issue-PR pairs | gold patch gives F2P > 0 and P2F = 0 | not stated | "tens of thousands" of environments |
| GLM-5 SWE14 | issue-PR pairs, RepoLaunch setup | F2P and P2P tests extracted by LLM-written log parsers | not stated | over 10k environments, 9 languages |
| V3.2 general agent16 | LLM-written data, tools, task, verifier | tools-only solution must pass the verifier | iterated harder; keep pass@100 > 0 | 1,827 environments; 4,417 tasks |
| GLM-5 terminal14 | seed tasks or web pages, Harbor format | refine agent, manual rubrics; Harbor validation | stratified sampling (web route) | thousands; build accuracy over 90% |
| GLM-5 search14 | web knowledge graph, over 2M pages | bidirectional question-answer verification | drop if solved by a tool-free model (1 of 8) or an early agent | not stated |
| Kimi K315 | knowledge graph; mock applications; AET | independent verifier; per-event criteria | hidden verifiers on held-out scenarios (AET) | not stated in the sections read |
| Kimi K2 (SFT data)7 | 3,000+ MCP tools; 20,000+ synthetic tools | LLM judge vs per-task rubric; real sandboxes for code | not stated | "tens of thousands" of training examples |
6.4 What a pass@100 > 0 filter admits
DeepSeek-V3.2 keeps only synthesized tasks with non-zero pass@100 after RL with V3.216. That is a solvability filter, and its arithmetic is worth doing because it is easy to read as a difficulty filter. Let \(p\) be a task's per-sample success probability. The filter passes the task with probability \(1-(1-p)^{100}\), one half at \(p=1-0.5^{1/100}\approx0.0069\). A group of \(G\) samples gives a gradient only if it contains both outcomes (Module 10, Section 5; Module 11), with probability \(1-p^{G}-(1-p)^{G}\). At \(p=0.0069\) and \(G=16\), the group size Nemotron 3 Ultra uses11, that is 10.5% (derived): a task at the edge of the filter yields a usable group about one time in ten, and the filter says nothing about where learning is fastest.
What a pass@N filter admits, and what the group sees
Formulas: \(1-(1-p)^N\) and \(1-p^G-(1-p)^G\), independent samples, fixed per-task \(p\). N = 100 is the report's value; G = 16 is Nemotron 3 Ultra's11.
The arithmetic also shows what the filter catches. A verifier that wrongly rejects correct solutions drives \(p\) to zero and the task is dropped, a useful check. A verifier that wrongly accepts wrong answers raises \(p\) and passes easily, so the filter cannot catch false positives (my reasoning; the report does not discuss it). The check that does is a negative control: run a known-wrong or empty solution and require a reward of 0. None of the formats read here builds that in; R1's code pipeline approximates it by keeping test subsets that expose flaws in known-incorrect submissions (Module 10, Section 3.2).
6.5 The one environment ablation
V3.2's Section 4.3 is the only environment-design ablation in the reports read for this module (K2, K2.5, K3, V3.2, V4, GLM-5, Nemotron 3 Ultra); a search of their texts for "ablat" finds ablations of pre-training, RL infrastructure and distillation, none of environments, harnesses or difficulty filters. For difficulty, it samples 50 general synthesized tasks and evaluates the synthesis model and three frontier models (Table 7). For transfer, it runs RL on the SFT checkpoint using only synthetic general-agent tasks in non-thinking mode, against that SFT model and against V3.2-Exp, trained with RL only on search and code. RL on synthetic data "yields substantial improvements" on τ²-Bench, MCP-Mark and MCP-Universe, whereas "restricting RL to code and search scenarios does not improve performance on these benchmarks"16 (§4.3; the report's Figure 5 plots curves, which I do not quote numerically).
| Pass@K | DeepSeek-V3.2-Exp | Sonnet-4.5 | Gemini-3.0 Pro | GPT-5-Thinking |
|---|---|---|---|---|
| 1 | 12 | 34 | 51 | 62 |
| 2 | 18 | 47 | 65 | 75 |
| 4 | 26 | 62 | 74 | 82 |
Three limits. The sample is 50 tasks and one run per condition. The comparison arm differs in task content (code and search versus general agent), so the result says synthetic general-agent tasks add something code-and-search RL does not, not that synthetic beats real. And the generalization claim rests on the benchmarks' "environments and toolsets" not having been "encountered during RL training"16 (§4.1), an overlap the reader cannot audit without the environment lists. The task mix is also easy to misread: the report's "over 1,800 distinct environments and 85,000 complex prompts" are different populations. Table 1 of the report counts 24,667 code-agent tasks (real environment), 50,275 search-agent tasks (real environment, synthesized prompts), 4,417 general-agent tasks (synthesized environments) and 5,908 code-interpreter tasks (real), summing to 85,267 (derived). The 1,827 synthesized environments carry 5.2% of the tasks, about 2.4 tasks each (derived)16.
7. What to test before an environment feeds a gradient
The sources contain no end-to-end environment audit, so this section assembles one from checks that individual reports or libraries document, and says where a check has no documented source. Module 19 owns the exploit taxonomy and Module 20 the measurement questions.
7.1 Failures that are not the model's
An agent rollout touches containers, networks, simulated users and judges, and any can fail for reasons unrelated to the policy. GLM-5 is explicit: coding-agent sandboxes "may fail for reasons unrelated to the model (e.g., environment crashes)", and such failures "introduce noisy training signals because they reflect environment instability rather than the model's capability". It records the failure reason per sample, excludes environment failures, and repairs the resulting short GRPO group: "we pad the group by repeating valid samples if the number of valid samples exceeds half of the group size; otherwise, we drop the entire group"14 (§4.1.2). verifiers takes the same stance at the scoring layer: let judge failures propagate rather than score 0, and let a rubric judge return a drop verdict (for example rubric: outcome=task_defect) that "records no reward and drops the trace", so it "counts in no reward mean or training batch"4.
The repair rule has a simple cost curve. If each of \(G\) samples independently fails with probability \(q\), the group is dropped when at most \(G/2\) samples are valid: $$P_{\text{drop}}=\sum_{v=0}^{\lfloor G/2\rfloor}\binom{G}{v}(1-q)^{v}q^{G-v}.$$ With \(G=16\) and \(q=20\%\) (derived), 0.70% of groups are dropped but 96.5% need repair; with \(G=8\) and \(q=30\%\), 19.4% are dropped. Repeated samples carry extra weight in the group statistics, a side effect the report does not discuss (my reading). Independence is also generous: a broken task fails all its samples, so the dropped fraction then equals the fraction of broken tasks, whatever \(G\) is.
Group survival under environment failures
Rule: keep the group if more than G/2 samples are valid (GLM-514); formula as above. Independent failures are an assumption.
NeMo Gym addresses the same problem from the data side: v0.5.0 added gym eval reverify, which lets you "recompute rewards from stored rollouts without re-running inference", and v0.6.0 added health checks and token, tool-call, turn and latency diagnostics8. Re-scoring stored rollouts makes a verifier fix cheap and answers the versioning problem of Section 4.4.
7.2 Is the task itself valid
Validity checks in the sources fall into four kinds. A reference solution must score full reward: Harbor's oracle run10, verifiers' validate() hook for tasks that have "a known reference solution or gold check that can be verified without calling a model"4, and the gold-patch conditions of SWE-bench, SWE-Gym and V3.2192016. The answer must be unique and checkable: τ-bench writes instructions that admit one outcome3, and GLM-5's search pipeline rejects non-unique answers14. A capable policy must beat chance: TRL advises that "if a capable model can't score higher than a random baseline, the reward signal may need adjustment"2. And the pass-rate profile must sit in a learnable band, which Nemotron 3 Ultra handles by "reward profiling prior to training" and re-profiling when accuracy saturates11 (§3.2, §3.3.2; Module 10 Section 5 covers the filters).
Environment bugs also reach public benchmarks. GLM-5 reports Terminal-Bench 2.0 on a "verified" version that "resolves some ambiguous instructions" alongside the original14; DeepSeek-V4 "acknowledge[s] the environment-related issues noted by GLM-5.1", reports the original dataset for consistency, and gives approximately 72.0 for V4-Pro on the verified subset21 (§5.3.1). If a widely used benchmark needed a verified edition, a training environment built by a synthesis agent and checked by a self-verification loop is unlikely to be cleaner (my inference).
7.3 The checklist, with sources
| Check | What it catches | Documented in |
|---|---|---|
| Reference solution scores full reward | broken tasks, missing dependencies | Harbor oracle10; verifiers validate()4; V3.2 F2P and P2F16 |
| A wrong or empty solution scores zero | a verifier that is too lenient | none for environments; R1 test filtering (Module 10) |
| Solvable at least once, not always | unsolvable and trivial tasks | V3.2 pass@100 > 016; GLM-5 search filters14; K2 pass@k band7 |
| Unique, checkable answer | ambiguous tasks, multiple valid outcomes | τ-bench3; GLM-5 bidirectional check14 |
| A capable model beats random | sign errors, dead rewards | TRL2 |
| Infrastructure failures are excluded, not scored 0 | noise that looks like model error | GLM-5 rule14; verifiers4 |
| Truncation and timeout have a stated policy | length-driven reward noise | verifiers skip flags4; K2 penalty7; TRL total-length cap2 |
| The agent cannot reach the verifier | tampering, answer leakage | verifiers isolated scoring4; K3 hidden verifiers15 |
| Reward changes are versioned and rollouts re-scorable | silent drift of the objective | Gym versioning rule5; gym eval reverify8 |
| More than one harness per task distribution | overfitting to one tool schema or context rule | K315; Nemotron 3 Ultra11 |
Where experts disagree
The source map records no dispute about environment design, and this module's evidence explains why: reports describe their environments without ablating them. The one ablation, DeepSeek-V3.2's Section 4.3, compares synthetic general-agent RL against code-and-search RL and nothing else16. What exists is divergent practice with stated reasons and little head-to-head evidence. Four practices are separated below; whether to train under one harness or several has a stated position for diversity (Section 3.2) and none against, so it appears in the assessment.
Binary rewards on final state
TRL's OpenEnv guide (Wordle and Sudoku experiments)2; NeMo Gym's single-step tool verifier8; τ-bench3
A group-relative method uses only ranking inside a group, so partial credit adds a hackable target without adding rank information that a final-state check lacks: "binary rewards (1.0 for success, 0.0 otherwise) gave cleaner training signals than shaped rewards with partial credit"2. NeMo Gym's single-step tool verifier demands exact matches and scores a multi-call response 0, with the stated aim of "eliminating partial-credit reward hacking"8.
- Few places to hide a hack; easy to audit and re-score
- Consistent with the shaping theorem: non-potential dense rewards can change the optimal policy17
- An all-fail group carries no gradient, so hard tasks stall (Module 10)
- The supporting evidence is a practitioner remark with no numbers2
Graded or shaped rewards where progress is meaningful
Kimi K3's kernel and web tasks15; NeMo Gym's distance, milestone and per-step environments8; Ng et al. on safe shaping17
When the quantity is naturally continuous (speed against a roofline, pixel distance) or full success is rare, a graded reward gives signal that binary does not. K3 gates on correctness first, scales performance second, and adds a hack-detector15. Potential-based shaping is the one form with an optimality guarantee17.
- Ranks failures, so hard tasks still teach
- Matches the true objective where it is continuous
Real, executable environments first
SWE-Gym20; GLM-5's SWE pipeline14; Kimi K2's hybrid design7
Real repositories with real tests give "reliable reward signals" that were "largely absent" in software engineering20. K2 acknowledges "the inherent limitation of simulation fidelity" and adds real sandboxes for coding "where authenticity is crucial"7.
- Ground truth from tests the repository's authors wrote
- No simulator to drift from reality
- Needs a repository history; most agentic domains have none
- Environment setup cost: 200 human-hours for 2,438 instances in SWE-Gym20; 2026 reports automate it without publishing yields
Synthetic and simulated environments at scale
DeepSeek-V3.216; Kimi K3's mock applications15; Kimi K2's tool simulator7
Where the world cannot be snapshotted, build it: V3.2's synthetic general-agent tasks improved τ²-Bench, MCP-Mark and MCP-Universe where code-and-search RL did not16; K3's mock Gmail, Notion, Slack and Canvas offer "reproducible, large-scale interaction without external APIs or rate limits"15.
- Breadth independent of repositories or live services
- Verifier and state under the builder's control
- Fidelity is unmeasured; self-verification shows consistency, not relevance
- The one ablation is one run on 50 sampled tasks, against a non-matched arm16
Reward shape. Default to a binary reward on final state when a program can read an end state and the pass rate is mid-range. Use a graded reward when the quantity is natively continuous or pass rates are too low for groups to mix, and then do what K3 does: gate on correctness, scale second, and run a hack-detector you expect to extend; make dense per-step credit a potential over state. A matched-compute comparison of binary and partial credit across difficulty bands would settle it; Reasoning Gym has the difficulty knob to run one18, and no source reports it.
Real versus synthetic. The positions are complements: use real artifacts where they exist and synthesize where they do not, with the Section 7 checks on both. V3.2 shows synthetic tasks can transfer to unseen benchmarks, not that they match real tasks per unit of compute. A same-budget comparison of real and synthetic tasks in one domain, evaluated on environments in neither, would settle it.
One harness or several. K3 and Nemotron 3 Ultra argue for diversity1511, and V3.2's context rule shows a harness policy can change what is learned16. No report trains under one harness against several, so the case rests on mechanism (tool schemas and context rules are part of the task), not a measured gap. Training on one harness set and evaluating on a held-out harness would settle it. Until then, Nemotron's hedge is cheap: two or more harnesses per task distribution, at the cost of trace capture for each11.
What is changing
The interfaces are moving faster than the evidence about them. OpenEnv's README says the project "is in early development, so APIs may still change" and lists a technical committee of twelve organizations9. NeMo Gym shipped five minor releases in about four months, dated 06/04, 07/01, 08/06, 09/03 and 10/06/2026 in the README (read as month/day, consistent with the sequence), a mean gap of 31 days (derived)8. What is quoted here about those two, and about the main-branch documentation of verifiers and TRL, will drift within a quarter.
Three trends are visible. Interface convergence at the model boundary: five systems capture tokens there, three of them through a proxy or gateway (Table 3). The unit of environment is widening from a task to task plus harness plus runtime, with harness choice a configuration axis815. And construction is being automated end to end, with setup agents, refine agents and self-verification loops replacing SWE-Gym's 200 human-hours2014, which moves the failure mode from too few environments to too many unaudited ones. Open questions the sources leave: funnel yields for the 2026 pipelines, any ablation of harness diversity, a matched-compute comparison of reward shapes, the fidelity of simulated against real environments, and how reward changes made during a run are versioned.
References
- officialFarama Foundation. Basic Usage. Gymnasium documentation, live docs (no version stamp). https://gymnasium.farama.org/introduction/basic_usage/. Accessed 2026-10-27. Used for:
Envas an MDP class "not a perfect reconstruction", the reset and step return tuples, terminated versus truncated, and the Space types includingText. - officialHugging Face. OpenEnv Integration for Training LLMs with Environments. TRL documentation, main branch. https://huggingface.co/docs/trl/openenv. Accessed 2026-10-27. Used for: the
environment_factorycontract, tools versus environments, the reward-design tips (binary versus partial credit, check the final state, test the reward),max_completion_lengthover a whole conversation, server concurrency, white-box versus harness training. - paperYao, S., Shinn, N., Razavi, P., Narasimhan, K. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045v1, 17 Jun 2024. https://arxiv.org/pdf/2406.12045. Accessed 2026-10-27. Used for: the POMDP formulation and hidden database state (§2), the reward \(r_{\text{action}}\times r_{\text{output}}\) and its stated limitation, the simulated user, Table 1 statistics.
- repoPrime Intellect. verifiers documentation:
docs/architecture.md,docs/tasksets.md,docs/agent.md(main branch,verifiers.v1API, no version pinned). architecture.md, tasksets.md, agent.md. Accessed 2026-10-27. Used for: Taskset, Task and TaskData; the rollout lifecycle; shared versus isolated scoring; the interception server; skip-on-truncated and skip-on-timeout; infinite tasksets; reward weights;validate(); thedropverdict. - paperBrockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W. OpenAI Gym. arXiv:1606.01540v1, 5 Jun 2016. https://arxiv.org/pdf/1606.01540. Accessed 2026-10-27. Used for: the 2016 reset and four-element step interface, "environments, not agents", strict versioning of environments (§3).
- officialFarama Foundation. Env. Gymnasium API reference, live docs (no version stamp). https://gymnasium.farama.org/api/env/. Accessed 2026-10-27. Used for: the v0.26 split of
doneintoterminatedandtruncatedand its rationale,reset(seed, options), theinfofield, multi-agent pointer to PettingZoo. - paperKimi Team. Kimi K2: Open Agentic Intelligence. arXiv:2507.20534v2, 3 Feb 2026. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-27. Used for: §3.1.1 agentic data synthesis (SFT), §3.2.1 verifiable-reward gym and sandbox scale, §3.2.3 budget control, §3.3.4 agentic rollout.
- repoNVIDIA. NeMo Gym (NVIDIA-NeMo/Gym README). README on main showing release v0.7.0 dated 10/06/2026 (date format unlabeled; read as month/day). https://raw.githubusercontent.com/NVIDIA-NeMo/Gym/main/README.md. Accessed 2026-10-27. Used for: the environment definition, quick-start servers, release notes v0.3.0 to v0.7.0, the environment table (226 rows parsed by me) and the reward shapes in it.
- repoMeta-PyTorch with Hugging Face and partners. OpenEnv: Agentic Execution Environments (README). README on main, BSD 3-Clause, "early development". https://raw.githubusercontent.com/meta-pytorch/OpenEnv/main/README.md. Accessed 2026-10-27. Used for:
reset,step,stateover WebSocket, Docker per environment, white-box versus loop-owning training modes, HarborTrainingTrace, the maturity statement and the technical committee. - officialHarbor Framework. Create a task (Harbor tutorial). https://docs.harborframework.com/tutorials/create-a-task. Accessed 2026-10-27. Used for: the task directory layout,
task.tomltimeouts, the reward file, the oracle-agent validation run, RewardKit pointer. - paperNVIDIA. Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv:2606.15007v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-27. Used for: §3.2 unified RLVR, harness diversity, 16 rollouts per sample and reward profiling; §3.3 single-turn rollouts; Appendix A.2 harness robustness.
- repoNVIDIA. NVIDIA Nemotron Developer Repository (NVIDIA-NeMo/Nemotron README). README on main, Apache 2.0; Nemotron 3 Ultra section. https://raw.githubusercontent.com/NVIDIA-NeMo/Nemotron/main/README.md. Accessed 2026-10-27. Used for: the Ultra recipe scope (Pretrain → SFT), the "Open-Source Data Only" notice and its statement that results differ from the tech report benchmarks.
- paperKimi Team. Kimi K2.5: Visual Agentic Intelligence. arXiv:2602.02276v2. https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-27. Used for: Appendix D, the Gym-like interface, white-box and black-box environments, LLM Gateway, token-in-token-out.
- paperGLM-5 Team (Zhipu AI and Tsinghua University). GLM-5: from Vibe Coding to Agentic Engineering. arXiv:2602.15763v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-27. Used for: §4.1.2 TITO gateway and the environment-failure repair rule, §4.2.1-§4.2.3 SWE, terminal and search environment pipelines, the verified Terminal-Bench 2.0 note.
- paperKimi Team. Kimi K3: Open Frontier Intelligence. arXiv:2607.24653v2, 7 Aug 2026. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-27. Used for: §4.2.1 white-box harness, §4.2.2 knowledge-graph synthesis, §4.2.4 kernel tasks, §4.2.5 mock applications, §4.2.6 Autonomous Execution Tasks, §4.2.7 web development.
- paperDeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556v1, 2 Dec 2025. https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-27. Used for: §3.2.1 context management, §3.2.3 Table 1 and the code and general-agent environment pipelines, §4.1 generalization claim, §4.3 Table 5 and Figure 5.
- paperNg, A. Y., Harada, D., Russell, S. Policy invariance under reward transformations: Theory and application to reward shaping. ICML 1999. https://people.eecs.berkeley.edu/~pabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf. Accessed 2026-10-27. Used for: Theorem 1 (§3) on potential-based shaping and the bicycle and soccer examples (§1); read from the PDF pages as images.
- paperStojanovski, Z., Stanley, O., Sharratt, J., Jones, R., Adefioye, A., Kaddour, J., Köpf, A. Reasoning Gym: Reasoning Environments for Reinforcement Learning with Verifiable Rewards. NeurIPS 2025 Datasets and Benchmarks; arXiv:2505.24760v2, 20 Oct 2025. https://arxiv.org/pdf/2505.24760v2. Accessed 2026-10-27. Used for: design principles P1-P3 (§2), zero-shot scores and the difficulty cliff (§3, Figure 3), curriculum experiment and Table 5 (§5), limitations (§7).
- paperJimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024; arXiv:2310.06770. https://arxiv.org/pdf/2310.06770. Accessed 2026-10-27. Used for: the three-stage construction pipeline and resolution criterion (§2), Table 10 (93,139, 11,407, 2,294), execution-based validation (Appendix A.3).
- paperPan, J., Wang, X., Neubig, G., Jaitly, N., Ji, H., Suhr, A., Zhang, Y. Training Software Engineering Agents and Verifiers with SWE-Gym. arXiv:2412.21139v2, 6 Jun 2025. https://arxiv.org/pdf/2412.21139v2. Accessed 2026-10-27. Used for: §3 and §3.1, the 2,438 validated instances, 64,689 raw instances, 200 human annotation hours, 10,000 CPU core hours, 6 TB of images.
- paperDeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348v1. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-27. Used for: the Terminal-Bench 2.0 environment-issue note and verified-subset score (§5.3.1).