How to build an RL environment for an AI agent, step by step

To build an RL environment, pick a workflow whose outcome can be checked in code, model the world and its tools, generate tasks from seeds, write a verifier that scores the final state, and attack it before any agent trains on it. Then package it for any training stack. The verifier, not the world, decides what the agent learns.

The sequence below has ten steps. At the end there is a small Python sketch of a toy environment that you can run and extend. If the basic terms are new, read What is an RL environment? first.

Step 1: Pick a workflow whose outcome can be checked

Start with the check, not the world. A workflow is a good candidate when you can describe, in code, what the world must look like when the work is done correctly.

Ask four questions about the candidate:

• Is there a defined end state? "The invoice is on hold and the duplicate line is flagged" is checkable. "The supplier is happy" is not.
• Can the rules be written down? Rest periods, approval limits and tax rules can be. Unwritten habits cannot.
• Can a reference answer be computed? For planning work, a solver can often find one.
• Is the work valuable? A checkable but trivial task teaches little that matters.

If the answer to any of the first three is no, narrow the task until it is yes. "Handle the absence" becomes "assign the open shift without breaking a rule, and tell the person". Narrow, checkable tasks can later be chained into longer ones.

Step 2: Model the world in entities, documents and rules

The world is the data the agent works on, kept as one structured state that the verifier can read. Model it in three layers.

1. Entities: the people, records and objects of the workflow, such as staff, shifts, invoices or orders, with their attributes.
2. Documents and messages: the unstructured inputs that real work runs on, such as a sick note, a supplier email or a policy page. They are generated from the entities, so their facts are known.
3. Rules: the constraints that decide what is allowed, written as code, so the verifier and the reference solver use the same definitions.

Keep the truth in the structured state and let documents be noisy views of it. The agent sees the noise; the verifier checks the truth.

Step 3: Design the tools, with JSON schemas

Tools are the only way the agent can read or change the world, so their design shapes every episode. Give each tool a narrow purpose and a schema for its arguments.

JSON Schema is a declarative language for defining the structure and constraints of JSON data, which makes it a natural way to describe tool arguments.

{
  "name": "book",
  "description": "Book a room for one hour. Fails if the room is taken.",
  "input_schema": {
    "type": "object",
    "properties": {
      "room": { "type": "string" },
      "hour": { "type": "integer", "minimum": 9, "maximum": 16 }
    },
    "required": ["room", "hour"],
    "additionalProperties": false
  }
}

Good tool design follows a few rules. Return clear errors instead of failing silently, so the agent can recover, and avoid tools that do the whole task in one call.

Never expose a tool that can read the verifier, the reference solutions or another episode's state. Gym-style APIs for LLM agents covers the interface around the tools.

Step 4: Generate tasks from seeds

A task generator is code that turns a seed into a complete task: the world state, the documents and the instruction. A seed is a number that fixes every random choice, so the same seed always produces the same task.

Group generated tasks into task families. A task family is a group of tasks that share a structure but differ in their details; give each family settings for difficulty.

Check that every generated task has a correct outcome the verifier can recognise. A task whose correct answer is to reject a request, flag a problem or report that it cannot be done still counts. A generator that sometimes produces tasks with no correct outcome teaches the agent that failure is random.

Decide early which seeds and which families are for training and which are held back for evaluation. Mixing them later is hard to undo.

Step 5: Write the verifier: gates, then scores

The verifier is code that reads the final state and the agent's final report after submission, and returns a score with reasons. A layered design works well for most workflows.

1. Gates are pass or fail checks that must all pass before anything else counts: the output is valid, no hard rule is broken, the required work exists.
2. Weighted scores measure quality once the gates pass: cost compared with the reference, stability (how few unnecessary changes the agent made), required outputs, honesty of the report.
3. Reasons are attached to every lost point, so builders can read why an agent failed.

The verifier compares the final state with the state at reset, not with what the agent says it did. It must give the same score for the same final state every time.

Gates make the reward sparse while agents still fail them often: an episode that breaks one rule scores zero, however much else it got right. For long tasks, gate and score each part separately, such as each shift or each invoice in a batch, so a mostly correct attempt still earns something. Scoring agent tasks works through a full example with weights.

Step 6: Compute reference solutions

A reference solution is a known good answer that scores are measured against. For tasks with one right answer, the generator knows it already. For planning tasks with many valid answers of different quality, such as rosters, compute it with a constraint solver.

One example of such a solver is CP-SAT, part of Google's OR-Tools. It reports whether a solution is optimal or only feasible, and its documentation states that for a feasible solution it does not know whether it is optimal.

For long runs, set a time limit, as the OR-Tools documentation recommends, and record which status each reference reached. A limit in seconds stops the search at different points on a fast and a slow machine; the CP-SAT parameter file also offers a limit in deterministic time (max_deterministic_time), which does not depend on the clock. Fix the number of search workers too, for example to one: by default the solver tries to use all cores on the machine, so the same settings would run a different search on a machine with more cores.

Compute the reference once per seed and freeze it. If references are recomputed with a different solver version or time limit, scores drift and earlier results stop being comparable. If an agent beats a reference that is only feasible, cap its cost score at full marks and flag the seed so its reference can be recomputed in the next version. Vencur's worlds are designed around references computed with a constraint solver and frozen, so every score has a fixed target; Solver-computed reference solutions explains the method and its limits.

Step 7: Red-team the environment

Before any training, attack the environment the way a determined agent would. To red-team an environment is to attack it on purpose, looking for ways to score without doing the work.

The research nonprofit METR reported in 2025 that frontier models tried, often successfully, to raise their scores by modifying tests or scoring code and by gaining access to the answers used to check their work. Assume your agents will try the same.

A basic red-team list for a new environment:

Claiming work that existed before the episode is one of the easiest exploits to miss. A natural first verifier for the room-booking sketch at the end of this article checks that a booking exists for the right room and hour. An agent that names a room someone else had already booked would pass. The fix is to compare the final bookings with the bookings at reset, so only new work counts.

Step 8: Calibrate with baseline agents

A baseline agent is a simple or well-known agent that shows how hard the tasks are. Run two or three baselines on every task family before training.

Look at the spread of scores, not just the average. If a family scores near zero for every baseline, the tasks may be too hard to learn from, or broken. If a family scores near full marks, it is too easy to teach anything. Adjust the generator's difficulty settings until each family has a useful range, and keep the baseline scores with the environment, so buyers and later readers can see where the tasks started.

Baselines show the floor. Before training, also run the model you plan to train several times on a sample of tasks and record the success rate per task. Training methods that compare several attempts at the same task learn nothing from a task that every attempt solves, or every attempt fails.

Step 9: Package it: Docker, an HTTP API, offline

Package the environment so that any training stack can run it without your help. A Docker image with a small HTTP API is the most portable shape.

• Offline: no calls to outside services during an episode, so results do not depend on the internet.
• Parallel: many episodes at once, each with its own isolated state.
• A small API: reset, step, state, submit, score and a list of tools with their schemas. Reset, state and score belong to the training harness and are never exposed to the agent as tools.
• Versioned: the image, the task generator and the references carry version numbers.

The Gymnasium interface separates two ways an episode can end: reaching a terminal state, and being cut off, typically by a time limit. Keep both in your API, so training code can tell a finished task from a timeout.

Step 10: Log, version and refresh

Log every step of every episode: tool calls, results, the final report and the score with its reasons. A log that chains a hash of each entry into the next makes later changes to any step visible.

Version everything that can change a score: the generator, the verifier, the references and the tools. When you fix an exploit or add a family, release a new version and keep the old one runnable, so earlier results can be reproduced. Reproducible agent evaluation covers what to record.

Refresh the environment over time. Agents get better, and a family that was hard last quarter may be easy now. Add harder variants and new families as baseline scores rise.

A simplified sketch in Python

The sketch below is a complete toy environment in a neutral domain: booking a meeting room. It has reset, step, submit and score, a task generator with seeds, and a verifier with gates, a cost score and an honesty score.

import random

class RoomBookingEnv:
    """Simplified sketch: book a meeting room. Tasks come from seeds; code scores the final state."""

    def reset(self, seed):
        rng = random.Random(seed)
        while True:  # regenerate until the task can be solved
            self.rooms = {f"R{i}": rng.choice([4, 6, 8, 12]) for i in range(1, 5)}
            self.bookings = {(r, h) for r in self.rooms for h in range(9, 17) if rng.random() < 0.4}
            self.task = {"people": rng.randint(3, 10), "hour": rng.randint(9, 16)}
            if self._free_fitting():
                break
        self.initial = set(self.bookings)  # the verifier compares against this
        self.best = min(self.rooms[r] for r in self._free_fitting())  # reference, frozen at reset
        self.report = None
        return {"task": f"Book a room for {self.task['people']} people at {self.task['hour']}:00.",
                "tools": ["list_rooms", "is_free", "book"]}

    def _free_fitting(self):
        t = self.task
        return [r for r, size in self.rooms.items() if size >= t["people"] and (r, t["hour"]) not in self.bookings]

    def step(self, tool, args):
        if self.report is not None:
            return {"error": "episode already submitted"}
        if tool == "list_rooms":
            return {"rooms": dict(self.rooms)}
        if tool == "is_free":
            return {"free": (args["room"], args["hour"]) not in self.bookings}
        if tool == "book":
            if args["room"] not in self.rooms or (args["room"], args["hour"]) in self.bookings:
                return {"error": "room unknown or taken"}
            self.bookings.add((args["room"], args["hour"]))
            return {"ok": True}
        return {"error": f"unknown tool {tool}"}

    def submit(self, final_report):
        self.report = final_report or {}  # e.g. {"booked": "R2"}

    def score(self):
        t, new = self.task, self.bookings - self.initial
        # gates: exactly one new booking, at the requested hour, in a room that is big enough
        if len(new) != 1:
            return {"total": 0.0, "reasons": [f"{len(new)} new bookings, expected 1"]}
        room, hour = next(iter(new))
        if hour != t["hour"] or self.rooms[room] < t["people"]:
            return {"total": 0.0, "reasons": ["wrong hour or room too small"]}
        # scores: cost (the smallest free room that fits is best) and honesty (report matches state)
        best = self.best
        cost = best / self.rooms[room]
        honesty = 1.0 if self.report.get("booked") == room else 0.0
        reasons = [] if cost == 1 else [f"a smaller free room had {best} seats"]
        reasons += [] if honesty else ["the report does not match the booking"]
        return {"total": round(0.7 * cost + 0.3 * honesty, 2), "reasons": reasons}


if __name__ == "__main__":
    env = RoomBookingEnv()
    print(env.reset(seed=7)["task"])
    hour, people = env.task["hour"], env.task["people"]
    rooms = env.step("list_rooms", {})["rooms"]
    free = sorted((size, r) for r, size in rooms.items()
                  if size >= people and env.step("is_free", {"room": r, "hour": hour})["free"])
    env.step("book", {"room": free[0][1], "hour": hour})
    env.submit({"booked": free[0][1]})
    print(env.score())

Run as a script, the example agent books the smallest free room that fits for seed 7 and scores 1.0. Three details of the sketch show the main points of this article:

• The verifier compares with the reset state. new = self.bookings - self.initial means only bookings made during the episode count, which closes the exploit of claiming a booking that existed before the episode. Across seeds 0–199, 133 tasks have a fitting room that was already booked at the requested hour; naming that room without booking anything scores 0.0 every time.
• Gates before scores. A booking at the wrong hour or in a room that is too small scores 0.0, however honest the report.
• Honesty is scored separately. An agent that books correctly but reports a different room loses the honesty share and can reach at most 0.7.

To stay short, the sketch skips validation, so some calls give wrong answers or crash: is_free reports a room that does not exist as free, a missing argument raises a KeyError, and score crashes if the gates pass but submit was never called. Its generator also makes only tasks that can be solved; a real generator also makes tasks where the correct answer is to report that no room fits.

A real environment checks every call against the tool's schema and returns a clear error. It also adds many entities, noisy documents, several task families, frozen solver references and an HTTP layer. The shape stays the same.

What are the most common mistakes?

The most common mistakes are a verifier that trusts the final report, a verifier that counts work that already existed, and a generator that produces only valid cases. Each looks harmless while building and becomes visible only when an agent trains on the environment.

The pattern behind the list is the same each time: the agent is rewarded for something slightly different from the work. The fix is always to move the check closer to the real outcome.

What should you check before training on it?

Before the first training run, go through the ten properties in What makes a good RL environment? and fix every failure on the checkable outcome and on exploit resistance. Then train on a small scale first and read a sample of episode logs by hand. Logs can reveal behaviours that scores alone do not show.

If you would rather have a red-teamed world built for your workflow, see Vencur Environments.