Glossary: RL environments and robot learning

AI agent

An AI system that works towards a goal over several steps: it decides on an action, such as calling a tool, looks at the result and decides what to do next. In reinforcement learning, the agent is the learner and decision maker.

Related: Policy, Tool call, RL environment

Baseline agent

A simple or well-known agent that is run on an environment to show how hard its tasks are. Baseline scores make difficulty visible and show whether a new agent really does better.

Related: Curriculum, Evaluation harness

Behaviour cloning

The most direct form of imitation learning: a policy is trained to replicate the actions an expert took in each recorded situation.

Benchmark contamination

When test items from a benchmark, or close copies of them, end up in a model's training data. The model's score then overstates how well it handles problems it has not seen.

Related: Seed, Evaluation harness

Constraint solver

Software that searches a very large set of candidate solutions for ones that satisfy all given constraints, and, if asked, for the best one under an objective such as cost.

CP-SAT

The primary constraint-programming solver in Google's open-source OR-Tools, designed for problems with integer variables such as scheduling and assignment. It reports whether a solution was proven optimal or is only known to be feasible.

Curriculum

An order of training tasks from easier to harder, so a learner first masters simple cases and then builds on them.

Related: Baseline agent, Task family

Deterministic verifier

A verifier that always gives the same score for the same final state, because it uses code and fixed references instead of a model's judgement.

Egocentric video

Video recorded from the point of view of the person doing an activity, with a camera they wear, usually on the head. The hands and the objects they use stay in view.

Embodiment gap

The difference between the body that produced the training data and the robot that has to act, for example a human hand and a two-finger gripper. Actions recorded with one body do not transfer directly to another.

Episode

One complete attempt by an agent at one task, from reset to submission.

Episode log (hash-chained)

A record of every step in an episode in which each entry includes a hash of the previous one, so a later change to any step becomes visible, as long as the latest hash is stored where the log cannot be rewritten.

Related: Episode, Trajectory

Evaluation harness

The code that runs an agent on a set of tasks, records every episode and computes the scores the same way every time.

Exclusive licence

A licence under which the rights to a world or dataset go to one licensee only, for the agreed scope and period. A non-exclusive licence allows the same material to be licensed to others.

Gate

A pass/fail check that must pass before any score counts, such as "every shift is covered".

Related: Verifier, Partial credit

Ground truth

The correct answer or state that a result is compared against. In an RL environment it is often a reference solution or the true state of the simulated world.

Related: Reference solution, Verifier

Gym API

The interface of OpenAI's Gym library, continued by Gymnasium: reset starts an episode and returns the first observation, and step applies an action and returns the next observation, a reward and whether the episode has ended.

Related: Observation, Reward, Episode

Hand keypoints

Points that mark the joints and fingertips of a hand in an image, such as the 21 landmarks found by Google's MediaPipe hand model. Tracked over time, they show how a hand moves.

Honesty check

A check that compares what an agent claims in its final report with the actual final state, and takes points away for false claims and for required items the report leaves out.

Related: Reward hacking, Verifier

Imitation learning

Learning how to act from demonstrations by an expert, instead of from rewards alone.

LLM-as-a-judge

Using a large language model to grade or compare outputs, often with a rubric. Studies have documented biases such as favouring an answer because of its position or its length.

Long-horizon task

A task that needs many steps before the result can be checked, so a mistake early on may only show up much later.

Model Context Protocol (MCP)

An open protocol for connecting AI applications to external tools and data. Servers describe their tools with a name and a JSON Schema for the inputs, and clients call them with JSON-RPC messages.

Related: Tool call, Gym API

Observation

What the agent receives from the environment after each action, such as text, a screenshot or a list of values. It may show only part of the full state.

Related: State, Gym API

Outcome reward

A reward given for the final result only, such as whether the answer is right or the end state is correct.

Related: Process reward, Reward, Gate

Partial credit

Points for a result that is partly right, with a reason for every point lost, instead of a single pass or fail.

Related: Gate, Verifier

pass@k and pass^k

pass@k is the chance that at least one of k attempts at a task succeeds; pass^k is the chance that all k attempts succeed. pass@k shows whether an agent can solve a task at all, pass^k how reliably it does so.

Policy

The agent's way of choosing actions: a mapping from what it perceives to the actions it takes.

Post-training

The training a language model receives after pre-training, such as supervised fine-tuning on demonstrations, learning from human preferences and reinforcement learning with verifiable rewards.

Process reward

A reward for each intermediate step of a solution, not only for the final result.

Red-teaming (of environments)

Attacking your own environment the way a determined agent would, for example by editing tests, reading hidden answers or claiming work that was not done, and closing every gap found before agents train on it.

Reference solution

A known good solution to a task, used as the target when scoring. At Vencur, references are computed with a solver and frozen.

Reinforcement learning (RL)

Learning what to do by trial and error: an agent acts, receives numerical rewards and adjusts its choices to collect as much reward as possible over time.

Related: Reward, Policy, RL environment

Reward

The number an environment returns to tell the agent how well it did. In reinforcement learning, the agent learns to collect as much reward as possible over time.

Reward hacking

When an agent earns a high reward without doing the intended task, for example by exploiting a flaw in the scoring. Frontier models have been observed modifying tests or scoring code to raise their scores.

Reward model

A model trained to predict which outputs people would prefer, used as the reward signal when tuning a language model. Because it is learned, it can be gamed, which is one reason to prefer verifiable checks where they exist.

RL environment

A software world in which an AI agent acts through tools and code scores the outcome. Agents learn from many scored attempts, called episodes.

RLHF

Reinforcement learning from human feedback: people compare or rank a model's outputs, a reward model learns to predict their preferences, and the language model is then tuned with reinforcement learning to score well on that reward model.

RLVR

Reinforcement learning with verifiable rewards: the reward comes from a check that can be computed, such as whether a maths answer is correct or code passes its tests, instead of from a learned reward model. The term was introduced with the Tulu 3 post-training recipe in 2024.

Rollout

One run of a policy in an environment, from the first observation to the end, recorded so it can be scored or used for training.

Related: Episode, Trajectory

Rubric

A written list of criteria and points used to grade work, by people or by a model. Rubrics suit qualities such as tone, which code cannot check exactly.

Related: LLM-as-a-judge, Partial credit

Sandbox

An isolated environment in which an agent's actions and code run without touching real systems, so episodes are safe and can be reset.

Related: Evaluation harness, Episode

Seed

A number that fixes all random choices when a task is generated, so the same seed always produces the same task.

Sim-to-real gap

The difference between how a system behaves in simulation and in the real world, which can make a policy trained in simulation fail on real hardware. It is also called the reality gap.

Related: Embodiment gap, World model

Specification gaming

Behaviour that satisfies the literal specification of an objective without achieving the intended outcome. Reward hacking is one form of it.

State

The full situation of the environment at one point in time, including everything the scoring may depend on. The agent usually sees only part of it, through observations.

Related: Observation, Trajectory

Supervised fine-tuning (SFT)

Training a pre-trained model on examples of the desired output, such as answers written by people, so that it learns to produce similar outputs.

Task family

A group of tasks that share a structure but differ in their details, generated from different seeds.

Related: Seed, World (Vencur's term)

Teleoperation

Controlling a robot remotely, often through a matching controller moved by a person. It is a common way to record demonstrations that contain the robot's own actions.

Tool call

An action in which an agent asks for a function to be run, by name and with structured arguments, and then receives the result.

Trajectory

The sequence of states, actions and rewards produced during an episode. For a language agent, it is the record of messages, tool calls and results.

Verifier

The code that decides whether an agent's work is correct and how many points it earns.

Vision-language-action (VLA) model

A model that takes camera images and a language instruction and outputs robot actions, for example by expressing the actions as tokens, like text.

Related: World model, Egocentric video

World (Vencur's term)

A complete simulated workplace packaged as one RL environment: its data, documents, people, rules, tools, tasks and verifier.

Related: RL environment, Task family

World model

A learned model of how an environment behaves, for example a compressed representation that predicts what happens next. An agent can plan with it or even be trained inside it.