RL environments vs. human-labelled data: which does your agent need?

Use human-labelled data to teach format, style and judgement that code cannot check, and use RL environments with verifiable rewards when the result can be checked in code and the work spans many steps. Published recipes such as InstructGPT and Tulu 3 use more than one kind, in stages. The deciding question is what the model should learn.

This article compares the three main kinds of post-training data, explains what each teaches well and badly, and ends with a decision table. For the basics of environments themselves, start with What is an RL environment?.

What are the three kinds of post-training data?

Post-training data comes in three main kinds: demonstrations, preferences and environments. Each produces a different training signal.

Demonstrations are examples of the desired output, usually written or recorded by people. Supervised fine-tuning (SFT) is training a model to reproduce such example outputs, token by token. In the reinforcement learning textbook by Sutton and Barto, supervised learning is learning from labelled examples provided by a knowledgeable external supervisor.

Preferences are comparisons: people see several outputs for the same prompt and rank them. Reinforcement learning from human feedback (RLHF) is training a model against a reward model learned from such rankings. In the InstructGPT work, Ouyang et al. (2022) trained a reward model to predict which output labellers would prefer, then used it as the reward function for reinforcement learning. Direct Preference Optimization (DPO) is a method that trains directly on preference pairs, without first training a separate reward model; the Tulu 3 report groups it with "direct alignment algorithms, which learn directly from the data".

Environments with verifiable rewards are interactive worlds where the model acts and code checks the result. Reinforcement learning with verifiable rewards (RLVR) is reinforcement learning in which the model is rewarded only when code verifies its output as correct. The name comes from the Tulu 3 recipe, which applied it to single-turn tasks such as maths and precise instruction following; environments apply the same idea to the final state of a world.

What does the same task look like in each kind of data?

As a demonstration, a task is a recorded sequence of steps; as a preference pair, it is two outputs with a person's choice between them; as an environment task, it is a generated world plus a code check on the final state. One invoice task shows all three.

The demonstration captures one good path but says nothing about what happens if the agent approves the invoice by mistake. The preference pair captures the quality of the note but not whether the hold was correct. The environment captures exactly what its verifier checks, for thousands of invoices, but in this example nothing checks the content or tone of the note. A complete training plan usually needs more than one of these.

What does each kind teach well, and what does it teach badly?

Demonstrations teach format and style but not recovery from mistakes; preferences teach judgement about quality but can be gamed through the reward model; environments teach correct outcomes, but only where code can check the result.

Demonstrations

Demonstrations are good at teaching shape: the format of a tool call, the structure of a report, the tone of a reply. They are also the most direct way to show a model a task it has never seen.

The weakness of demonstrations is that they show only the path the demonstrator took. A model trained on them never sees what happens after its own mistakes, so it learns little about recovering from them. The model's quality is also capped by the demonstrations: errors and shortcuts in the examples are learned along with everything else.

Preferences

Preferences are good at teaching judgement about quality where no exact answer exists: which explanation is clearer, which reply is more helpful, which summary leaves out less. Comparing two outputs needs no one who can write a perfect one.

The weakness of RLHF is the reward model in between. A learned model of preferences can be gamed: the model being trained finds outputs the reward model scores highly but people would not prefer. The DeepSeek-R1 authors chose not to use a neural reward model for DeepSeek-R1-Zero because it may suffer from reward hacking in large-scale reinforcement learning. DPO avoids a separate reward model, but it still learns only what the preference pairs contain.

Environments with verifiable rewards

Environments are good at teaching correct outcomes, especially in multi-step work. The agent tries, sees the consequences of its own actions, and gets a score that does not depend on how its answer sounds. It can find paths no demonstrator wrote down, although reinforcement learning with a pass or fail reward mainly strengthens behaviour the model already produces some of the time.

The weakness of environments is scope. They only work where the result can be checked in code, and building the world, the verifier and the task generator takes real engineering. A badly designed verifier teaches the agent to exploit it rather than to do the work.

When are exact checks possible?

Exact checks are possible when code can decide from the final state whether the result is correct, without any judgement of taste. Before choosing environments, test the task against four conditions.

1. The end state is defined. You can say precisely what the world should look like when the task is done, such as "every shift is covered" or "the invoice total matches the line items".
2. The rules can be written down. The constraints are known and stable enough to encode, such as rest periods or approval limits.
3. The end state can be tested directly, or a reference exists. A check such as "every shift is covered" needs no reference. Where quality is scored, such as cost, a known good answer or a solver-computed one gives the target.
4. Taste plays no part in passing. Style may still matter, but it does not decide whether the work is correct.

Code generation is the best-known case: Chen et al. (2021) evaluated generated programs by functional correctness, where a sample counts as correct if it passes a set of unit tests. Back-office work such as rostering, invoicing and approvals often meets all four conditions as well. Writing a persuasive email usually does not.

How do the three compare on cost and scale?

Demonstrations and preferences cost human time for every example. Environments cost engineering up front and then compute for every training run.

• Demonstrations scale with human effort. Each new example needs someone who can do the task well, and long, multi-step tasks take long to demonstrate.
• Preferences are often cheaper per example than full demonstrations, because judging an output can take less time than writing one. They still need a steady supply of human judgements, and the reward model may need retraining as the model being trained changes.
• Environments have a high fixed cost: the world, the tools, the task generator, the verifier and red-teaming against exploits. Scoring is cheap, but running the episodes is not: reinforcement learning samples many attempts per task, each a long trajectory of tool calls in its own copy of the world.

A new seed gives a new instance within a task family, a group of tasks that share a structure but differ in their details. A new kind of task needs a new family, which means new engineering. The variety of the generator limits what the agent can learn from it.

The cost difference shapes the decision. If you need a few hundred examples of a rare task, demonstrations are faster. If you need agents to practise the same kind of work thousands of times with exact feedback, an environment can pay for its fixed cost.

How do you combine the three kinds of data?

The common pattern is demonstrations first, then reinforcement learning, and the kinds can feed each other. Demonstrations give the model the basic format of the task; reinforcement learning then improves the outcomes.

The order matters for a practical reason. Reinforcement learning with a pass or fail reward learns nothing from a task the model always fails, or always passes. DeepSeek-R1, for example, uses Group Relative Policy Optimization (GRPO), which scores each attempt against the average of a group of attempts at the same task; if every attempt gets the same reward, no attempt stands out and there is nothing to learn. Demonstrations raise the success rate from zero, and tasks of calibrated difficulty keep it away from one.

Published recipes follow the demonstrations-then-reinforcement-learning pattern:

• InstructGPT fine-tuned on labeller demonstrations first, then trained a reward model on rankings and optimised against it.
• Tulu 3 lists supervised fine-tuning, Direct Preference Optimization and RLVR among its training algorithms.
• DeepSeek-R1 shows both the exception and the rule. DeepSeek-R1-Zero was trained with large-scale reinforcement learning without supervised fine-tuning first, but it struggled with poor readability and language mixing. DeepSeek-R1 therefore added a small amount of cold-start data before reinforcement learning: thousands of curated examples to fine-tune on first. R1-Zero's pass@1 on the AIME 2024 maths benchmark was already 15.6% when reinforcement learning started, so the base model produced some correct answers to reinforce; the cold-start data was added for readability and to improve reasoning further. Near the end of reinforcement learning, it created new supervised data through rejection sampling: keeping the model's own good outputs and training on them.

That last step shows how environments and demonstrations connect. An environment's verifier can filter an agent's own successful episodes, and those episodes can become demonstration data for the next round.

For agents doing operations work, a practical sequence looks like this:

1. Demonstrations of tool use and of the final report format, so the agent completes an episode often enough to earn rewards.
2. Reinforcement learning in environments on task families with exact checks, at a difficulty the agent sometimes but not always solves.
3. Preferences or rubrics for qualities code cannot check, such as the tone of messages, kept separate from the pass or fail decision.
4. Evaluation on held-out seeds and task families, so you measure improvement on tasks the agent has not seen.

Keep the exact checks in code throughout. If a model-based judge decides whether a hard rule was broken, the agent can learn to satisfy the judge instead of the rule. Deterministic verifiers vs. LLM-as-a-judge explains where each belongs.

Which does your agent need? A decision table

Your agent needs an environment when code can check the outcome of multi-step work, demonstrations when it must learn a new format, and preferences when quality is a matter of judgement. The table maps common goals to the data that teaches them; a gate is a hard check that sets the score to zero when it fails.

If most of your rows point to environments, the next step is deciding whether to build or buy them. What is an RL environment? covers the options, and RLVR, explained shows how business workflows become verifiable. Vencur builds RL environments for operations work; see Vencur Environments.