Gym-style APIs for LLM agents: reset, step, submit and score

A gym-style API for LLM agents keeps the classic reset and step calls, treats each tool call as an action and each tool result as an observation, and adds submit and score, so the reward arrives once, at the end, from a verifier. Over HTTP this becomes six calls: reset, step, state, submit, score and a list of tools.

This article explains the classic interface, what changes for language agents, who may call what, and how to expose the same tools through the Model Context Protocol. For the bigger picture, see What is an RL environment?.

What is the classic gym interface?

The classic interface is the Gym API of OpenAI's Gym library, which Gymnasium continues as a maintained fork: reset starts an episode, and step applies one action and reports what happened.

According to the Gymnasium API reference:

• reset(seed=...) returns the first observation and an info dictionary. The seed initialises the environment's random number generator, so a run can be repeated.
• step(action) returns five values: the next observation, the reward for the action, terminated, truncated and an info dictionary.
• terminated says whether the agent reached a terminal state of the task. truncated says whether the episode was cut off for a reason outside the task, typically a time limit.

The Basic Usage guide states the loop simply: if either flag is true, the episode ends. The two flags are kept apart because they mean different things for learning. Gymnasium's guide on handling time limits explains that methods which estimate future value should bootstrap after a truncation but not after a termination: a cut-off episode could have continued, a finished one could not.

What changes for LLM agents?

For language agents, the shape stays but the contents change. Actions become tool calls, observations become text, and the reward moves to the end of the episode.

Five changes deserve attention.

The agent decides when it is done. In many games the environment knows when the episode ends. In work tasks, the agent decides when the job is finished, so the API needs a submit call that ends the episode and records a structured final report.

The reward is episode-level. A half-edited roster or a half-checked invoice is neither right nor wrong, so it is usually better to return a reward of zero until submission. The cost is that on long tasks a single end reward makes it harder to tell which steps helped. Partial credit can still be computed at submission from the final state, per sub-task, and Scoring agent tasks shows how.

Errors are observations. When the agent calls a tool with bad arguments, the environment should return a clear error as the observation, not crash. Recovering from errors is part of the skill being trained.

One turn can hold several tool calls. Language models can request several tool calls in one turn. A simple rule is to run them as separate steps, in the order the model listed them, and return all their results to the model together before its next turn. Calls listed after submit in the same turn are rejected.

A turn can hold no tool call at all. If the model replies with plain text, return a short observation that asks for a tool call or a submission, and count the turn towards the step limit. Ending the episode there would punish a harmless habit; ignoring it would let an agent stall for free.

Who may call what?

The agent may list its tools, call them and submit; only the training harness may reset an episode, read its full state and read its score. If the agent could reach reset, state or score, it could read hidden information or its own score and learn to exploit them.

The verifier is the code that reads the final state after submission and computes the score. Serve the harness-only calls on a separate endpoint or behind separate credentials, so nothing the agent controls can reach them. The episode id belongs to the harness too: it binds each agent call to its episode, and the id never appears in a tool's arguments, so the model cannot forget it, invent one or pass another episode's id.

What does an HTTP version look like?

An HTTP version wraps the six calls in a service that runs in its own container, separate from the agent and the training code. The exchange below shows one short episode; the values are made up.

POST /reset                                   (harness)
{"task_id": "book-room", "seed": 900001}
→ {"episode": "ep_7f3a", "observation": "Book a room for 6 people at 16:00.",
   "versions": {"api": "1.2", "tools": "3.0", "world": "2026.10.1"}}

POST /step                                    (agent call, relayed with the episode id)
{"episode": "ep_7f3a", "tool": "list_rooms", "args": {}}
→ {"observation": {"rooms": {"R1": 6, "R2": 4, "R3": 8, "R4": 12}},
   "reward": 0.0, "terminated": false, "truncated": false}

POST /step
{"episode": "ep_7f3a", "tool": "book", "args": {"room": "R1", "hour": 16}}
→ {"observation": {"ok": true}, "reward": 0.0, "terminated": false, "truncated": false}

POST /submit                                  (agent's submit tool)
{"episode": "ep_7f3a", "final_report": {"booked": "R1"}}
→ {"terminated": true}

GET /score?episode=ep_7f3a                    (harness)
→ {"total": 1.0, "gates": {"valid": "pass"}, "reasons": []}

In the score, gates lists the pass or fail checks; a gate is a check that sets the whole score to zero when it fails.

A thin client turns these calls back into the familiar Gym loop:

• reset maps to POST /reset and returns the first observation.
• Each tool call maps to step, which returns the tool result, a reward of zero and both flags false.
• The submit tool is the final action: that step returns terminated=True and, as its reward, the verifier's total from GET /score.
• A step that hits a hidden limit returns truncated=True. What happens next depends on the method: with a value estimate, bootstrap from the last state; with methods that use one return per episode, either score the final state with the verifier, give a fixed failure reward, or drop the episode. Dropping biases batches and evaluation scores towards tasks the agent finishes.

Set limits in steps, tool calls or tokens, not in seconds: a limit in wall-clock time depends on server load, so the same actions could be cut off in one run and not in another. For language agents, running out of context length is another reason to truncate. How to build an RL environment contains a runnable Python version of the same toy task.

How should tool schemas be written?

Tool schemas should be strict and describe effects: required fields, no extra properties, enums and ranges where values are limited, and a description of what the tool does to the world. JSON Schema is a declarative language for defining the structure and constraints of JSON data, which is what a tool's arguments are.

Good schemas follow a few rules:

• Name tools after the action, such as assign_shift or put_invoice_on_hold, not after internal functions.
• Mark required fields and forbid extra ones, with required and "additionalProperties": false, so malformed calls fail loudly.
• Use enums and ranges where values are limited, such as shift codes or hours, so the schema documents the rules.
• Describe effects, not implementation. "Fails if the room is taken" helps the agent; "calls the booking service" does not.
• Return structured results. A result the agent can parse reliably is easier to act on than prose.

Keep the scoring out of the tools. No tool should return the reference answer, the verifier's view of the state or anything from another episode.

What should the final report look like?

The final report should be structured data with explicit claims, not free text. The verifier can then compare each claim with the final state, and an honesty score, a score for how far the report's claims match the state, becomes possible.

{
  "status": "complete",
  "assignments": [{"shift": "Tue night", "unit": "B", "staff": "W-01"}],
  "messages_sent": [{"to": "W-01"}],
  "open_issues": []
}

Three design choices make reports checkable:

1. Fixed field names and enums, such as a status of complete, partial or blocked, so there is nothing to interpret. The status is a claim like any other: the verifier checks it against the state.
2. Facts the state can confirm, such as assignments and messages, rather than opinions about the work.
3. A place for open issues, so an agent that could not finish has an honest way to say so.

Honesty checks explains how the comparison is scored.

How do you version an environment API?

Version the API, the tools and the world separately, and report all three in the response to reset. Training runs can be long, and a change in the middle of a run makes earlier and later episodes incomparable.

• API version: the shape of the six calls. Change it rarely and keep old versions running.
• Tool schema version: the list of tools and their arguments. Adding a tool keeps old clients working, but it changes what the agent can do and what its prompt lists, so it needs a new tool schema version, and scores from before and after are not directly comparable.
• World version: the task generator, the verifier and the frozen reference solutions, which are known good answers fixed per seed. Any change here can change scores, so it gets a new version number every time.

Record the three versions with every episode log. When a result is questioned months later, they are the first thing anyone needs.

When should you expose tools through MCP?

Expose the world's tools through the Model Context Protocol when the agent framework you train or evaluate already speaks MCP, or when the same tools should also work outside training. Keep episode control in a separate, small API.

The Model Context Protocol (MCP) is an open protocol that connects language model applications to external data and tools, using JSON-RPC 2.0 messages, according to its specification. Its tools section defines how a client lists a server's tools with tools/list and calls them with tools/call, and each tool declares its arguments as a JSON Schema in inputSchema. MCP's tool calls therefore match the agent-facing half of a gym-style API closely.

What MCP does not define, as of version 2026-07-28, is the rest of an RL episode: it has no reset, no submission, no reward and no seeds. The specification also notes that MCP has no protocol-level session, so a server cannot rely on implicit per-connection state to relate one tool call to the next. For stateful tools, its non-normative guidance suggests returning an explicit handle that the model carries forward as an argument. For training, binding the episode outside the model's arguments is safer, because a model that forgets, invents or swaps a handle would break the isolation of episodes. A design that works:

1. The training harness calls reset on the episode API and receives an episode id.
2. The harness starts an MCP endpoint for that episode, for example a server address or credential per episode, so every tool call reaches the right episode without the model naming it. The id is not part of any tool's inputSchema.
3. The agent finishes by calling a submit tool, which the harness relays to the episode API.
4. The harness reads the score from the episode API, whose harness-only calls the agent cannot reach.

How do you run parallel episodes and keep state isolated?

Give every episode its own id and its own copy of the world, created at reset from the seed. Training runs many episodes at once, and any shared state makes scores depend on what other agents did.

• One state per episode. Store each episode's world separately, keyed by its id, which the harness binds to every call.
• Complete resets. A reset rebuilds everything from the seed, including records the agent can only change indirectly.
• No shared caches with side effects. Read-only data, such as a policy text, can be shared; anything the agent can change cannot.
• Limits in steps, not seconds. Cut off episodes after a fixed number of steps, tool calls or tokens. A limit the task itself states, and that the agent can see, is part of the task, and reaching it is a termination; Gymnasium's guide treats a time limit inherent to the environment the same way. A cut-off the agent cannot see is a truncation.
• Deterministic tools. Avoid real clocks and random numbers that are not drawn from the episode's seed.

A quick test: replay a fixed sequence of actions for one seed alone, then replay the same actions for the same seed among a hundred parallel episodes. The states and the score must match exactly. Reproducible agent evaluation covers what else to record so that results can be repeated later.

A world is a simulated workplace with its tools, tasks and verifier. Vencur's worlds are designed around the same six calls, with a JSON schema for every tool; see Vencur Environments.