What is an RL environment? A practical guide for training AI agents

An RL environment is a software world in which an AI agent acts through tools while code scores the outcome. When the agent submits its work, a verifier reads the final state and turns it into a reward, and over many scored attempts the agent learns which behaviour earns higher scores.

This guide explains the idea from the ground up: what an environment is, the five parts of a training environment for language agents, how environments are used for training and evaluation, and why building a good one is harder than it looks.

What is an RL environment?

An RL environment is everything the agent interacts with, plus the rules that decide how well it did. A standard textbook on reinforcement learning, Sutton and Barto's Reinforcement Learning: An Introduction, calls the learner and decision maker the agent, and everything outside it the environment. The two interact in a loop: the agent acts, the environment responds with a new situation and a numerical reward.

For classic reinforcement learning, the environment might be a game or a simulated robot. The open-source Gymnasium library describes the interface most people know. reset starts an episode and returns the first observation and an info dictionary; step applies an action and returns the next observation, a reward, two flags and a dictionary of extra information. The flag terminated marks a natural end of the task, and truncated marks a cut-off such as a time limit.

For AI agents built on language models, the agent-specific version of the textbook definition is a software world in which the agent acts through tools while code scores the outcome. The loop looks different in practice:

• The actions are tool calls, such as reading a document, searching a list or updating a record.
• The observations are text: tool results, messages and error responses.
• The reward usually comes once, when the agent submits its work, from the verifier.

An episode is one complete attempt at one task, from reset to submission. A verifier is the code that reads the final state after submission and computes the score. An environment is useful when it can run thousands of episodes, each with a different task, and score every one of them the same way every time.

How is an RL environment different from a dataset?

A dataset is a fixed collection of examples; an environment is an interactive world that produces new experience every time an agent acts in it. That difference decides what a model can learn from each.

A dataset of demonstrations shows a model what good answers look like, and the model learns to imitate them. It never sees the consequences of its own mistakes, because the examples were recorded independently of it. In an environment, the agent is scored on the outcome it actually produced, including the results of its own mistakes, which a demonstration never shows.

Scoring the agent's own outcomes makes environments well suited to multi-step work, where one wrong step early on can break the result much later. It also makes environments more demanding to build, because the world has to react correctly to any sequence of actions, not just to the ones the author had in mind. RL environments vs. human-labelled data compares the options in detail.

What are the five parts of an RL environment?

A training environment for language agents can be described as five parts: a world state, tools, a task generator, a verifier and logs. If one is missing or weak, the agent learns the wrong lessons, or the scores stop meaning anything.

1. World state and data. The simulated workplace itself: records, documents, people, messages and the rules that apply. The state is the full truth the verifier will later check. A world is a complete RL environment for one workplace: its state, tools, tasks and verifier. When the agent messages a person in the world, the reply comes from scripted rules or from a language model. Cache or replay model-written replies when episodes must be reproducible; benchmarks such as τ-bench instead let the simulated user vary and measure how often all k trials succeed.
2. Tools and observations. The functions the agent may call, each with a clear input schema, and the results it gets back. Tools are the only way the agent can change the world.
3. Tasks generated from seeds. A task generator is code that creates many variations of the same kind of job. A seed is a number that fixes every random choice of the generator, so the same seed always produces the same task, and new seeds produce new ones.
4. The verifier. Code that reads the final state after submission and decides how many points the work earns, with a reason for every lost point. The verifier defines what the agent is actually rewarded for.
5. Episodes and logs. Every step of every episode is recorded: tool calls, results, the final report and the score. Logs make scores auditable and let builders read what agents actually did.

A fixed pool of tasks can also train a model: Tulu 3, for example, trained on fixed sets of maths problems. But a small pool gets memorised, and a seeded generator is a practical way to get many new, reproducible tasks at a difficulty you can control.

Tasks come in families

A task family is a group of tasks that share a structure but differ in their details. The family defines what kind of job it is; the seed decides the specifics.

Task families give the agent variety without losing control. Builders can hold some seeds back for evaluation, raise difficulty by changing generator settings, and add new families when agents master the old ones.

The verifier works in layers

A useful verifier does more than say pass or fail. One layered design checks hard gates first, such as "the output is valid" and "no hard rule is broken", and only then adds weighted scores for qualities like cost or stability. A gate is a check that sets the whole score to zero when it fails.

Gates stop an agent from collecting points for a polished but wrong result. Weighted scores then rank the results that pass every gate, so a valid but costly solution scores below a valid and cheap one. A reason attached to every lost point turns the score into feedback that builders can read. Scoring agent tasks explains this design with a worked example.

How are RL environments used to train and evaluate agents?

Environments serve two purposes: training, where scores become rewards that change the model, and evaluation, where scores measure the model without changing it. The same environment can do both if its tasks and verifier are designed for it.

Training. In post-training, a model attempts many tasks in the environment. A rollout is an episode run during training; each rollout produces a score. A reinforcement learning algorithm then updates the model so that behaviour leading to higher scores becomes more likely.

Tulu 3 used Proximal Policy Optimization (PPO) for this step; DeepSeek-R1 used Group Relative Policy Optimization (GRPO), which compares several attempts at the same task with each other.

Reinforcement learning with verifiable rewards (RLVR) is reinforcement learning in which the model receives a reward only when code verifies its output as correct, instead of being scored by a learned reward model. The Tulu 3 recipe (2024) gave the approach its name, and used it on maths problems and precise instruction following. The DeepSeek-R1 report describes a similar choice for its reasoning tasks, with rule-based rewards for correct answers and for a required output format in maths, code and logical reasoning; for general data, it also used reward models.

Environments for agents extend the same idea to multi-step work: the check is applied to the final state of a world after many tool calls. RLVR, explained covers this step in detail.

Evaluation. For evaluation, the task set is fixed and kept away from training data. The agent runs each task several times.

Because the verifier is deterministic code, the same final state always gets the same score; the agent's own sampling still varies, so results are reported over repeated runs with their spread. τ-bench, for example, proposes a metric, pass^k: the chance that all k independent trials of a task succeed, averaged across tasks.

What makes an environment useful for training, not only for testing?

An environment is useful for training when it produces a steady stream of tasks at the right difficulty, with feedback the agent can learn from. A good test set does not automatically meet that bar.

• Variety. Training consumes tasks quickly. A generator with many seeds and several task families keeps the agent from memorising answers.
• Calibrated difficulty. Difficulty has to be measured against the model being trained: useful tasks are ones it solves sometimes but not always. With a method such as GRPO, a group of attempts that all score the same gives no learning signal. Scores of simple baseline agents only show the floor.
• Informative scores. A single pass or fail at the end of a long task is a sparse signal. Scoring the parts of a task separately gives the agent more to learn from, as long as the gates still block wrong results.
• Speed and isolation. Training runs many episodes in parallel. Each episode needs its own copy of the world, so one agent's actions never leak into another's.
• Stable scoring. If the verifier or the reference answers change between runs, scores stop being comparable and training targets move.

Every Vencur world comes with tasks generated from seeds, scores of simple baseline agents and a hash-chained episode log, in which each entry carries a hash of the one before, so later changes become visible.

What kinds of RL environments exist?

Agent environments differ mainly in what the agent works on and how the result is checked. Public benchmarks show the range, from code to whole simulated companies.

SWE-bench draws its problems from GitHub issues and pull requests across popular Python repositories. WebArena builds fully working websites for e-commerce, forums, software development and content management. OSWorld runs agents in real operating systems; its 369 main tasks are defined and executed on Ubuntu. TheAgentCompany builds internal websites and data that mimic a small software company.

Back-office work such as payroll, e-invoicing or procurement combines many documents, rules and people, and its results can often be checked exactly. Vencur Environments focuses on this kind of work; worlds for these domains are Planned.

What does an episode look like in practice?

An episode is one attempt at one task: the agent reads its inputs, calls tools, changes the state and submits, and then the verifier scores the final state. The example below is invented to explain the idea and is not taken from any real system.

In this example, the verifier ignores the agent's wording, its reasoning and how confident it sounds. Only the end state and the honesty of the report count. The full walkthrough, with the tool calls, the score table and the log, is in Inside an episode.

What does the interface between agent and environment look like?

An agent environment can keep the classic shape of reset and step and add a separate submission and scoring step. Training code and agents can then drive any world the same way.

One HTTP version, the gym-style API that Vencur's worlds are built around, has six calls:

POST /reset    {"task_id": "...", "seed": 900001}
POST /step     {"tool": "...", "args": {...}}
GET  /state
POST /submit   {"final_report": {...}}
GET  /score
GET  /tools

reset starts an episode for a task and seed, and step runs one tool call. submit ends the episode with a structured final report, and score returns the verifier's result. tools lists every tool with a JSON schema for its arguments.

Only the training harness should be able to call reset, state and score. If the agent could call them, it could read hidden state or its own score and learn to exploit them, which is a path to reward hacking. The agent sees its tools and submits. Gym-style APIs for LLM agents explains each call and when to expose tools through the Model Context Protocol instead.

Why are good RL environments hard to build?

Good environments are hard because four goals pull in different directions: realism, exact checks, resistance to exploits and scale. Any one of them is manageable on its own; meeting all four at once is the actual work.

Realism. Real work is messy. Documents conflict, messages are vague, and rules have exceptions. An environment with clean, toy data teaches agents to expect clean inputs, and they fail on real ones.

Exact checks. In a verifiable environment, every task needs a verifier that decides correctness in code, so the outcome must be defined precisely. Where quality is scored, such as the cost of a roster, a good reference solution is needed to score against, and for planning tasks it often has to be computed with a solver. Solver-computed reference solutions explains how.

Resistance to exploits. Reward hacking is when an agent earns a high reward without doing the intended task, by exploiting a gap in the scoring. In 2025, the evaluation organisation METR reported frontier models trying, often successfully, to raise their scores by modifying tests or scoring code. An environment whose verifier can be reached, edited or fooled teaches reward hacking, so Vencur's practice is to attack its own verifiers with known exploits before a world ships.

Scale. Training needs many tasks with real variety, which is what the task generator and its families provide.

How to build an RL environment walks through the build sequence step by step.

Should you build, use open source or buy an RL environment?

Build, open source or buy depends on whether your domain already has a good environment and whether you need to train or only evaluate. The options can be combined.

• Use open source when a public environment covers your domain and you mainly need evaluation. Public benchmarks are well documented, but their tasks are public, so they may already appear in a model's training data.
• Build your own when the workflow is core to your product and you have engineers who know both the domain and the pitfalls of verification. Budget for the verifier, the task generator and red-teaming, not just the world.
• Buy when you need a new domain quickly, with exact checks and a clear licence, and you want the builder to carry the cost of red-teaming and refreshes.

If you buy, ask how success is verified, where reference solutions come from and which exploits were tested. Deterministic verifiers vs. LLM-as-a-judge explains why the scoring method matters so much.

In this series

• RL environments vs. human-labelled data: which does your agent need?
• What makes a good RL environment? 10 properties to check
• How to build an RL environment for an AI agent, step by step
• Gym-style APIs for LLM agents: reset, step, submit and score
• Buying RL environments: 12 questions to ask a vendor