Reinforcement learning with verifiable rewards (RLVR), explained

Reinforcement learning with verifiable rewards (RLVR) trains a model with rewards that come from a check that code can compute, such as whether a maths answer is correct or a program passes its tests, instead of from a learned reward model. In its original form, the model is rewarded only when its output is verified to be correct.

This guide explains where the idea comes from, what a training step looks like, how verifiers differ from reward models, and how the same principle extends from maths and code to multi-step business work. It ends with the limits of RLVR and how to design rewards that resist gaming.

What is RLVR?

RLVR is reinforcement learning in which the reward is computed by a check rather than predicted by a model. The model produces an output, and a verifier tests it. A verifier is the code that checks the output and turns the result into the reward.

The loop is the usual reinforcement learning loop: the model attempts many tasks, each attempt gets a reward, and the training algorithm makes rewarded behaviour more likely. What changes is the source of the reward. Instead of asking a model "how good is this?", RLVR asks code "is this correct?".

In the original Tulu 3 recipe, the reward is binary: a fixed amount for a verified output, zero otherwise. The gated and weighted rewards later in this guide extend that idea to graded scores, but every part of them is still computed by code. For an introduction to the environments these checks run in, see What is an RL environment?.

Where does the term RLVR come from?

The name comes from the Tulu 3 post-training recipe, published in 2024. Its authors introduced "a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR)".

In the Tulu 3 paper, RLVR samples outputs, verifies them with a deterministic function, gives a fixed reward if the answer is verifiably correct and zero otherwise, and trains against that reward with Proximal Policy Optimization (PPO). The authors applied it to tasks such as mathematics and precise instruction following, and describe it as a simplified form of existing approaches for bootstrapping the reasoning of language models.

Soon after, in January 2025, the DeepSeek-R1 report described training its R1-Zero model with an accuracy reward, which evaluates whether a response is correct, and a format reward, which enforces where the model writes its reasoning. For that model, the authors chose not to use a neural reward model, because it may suffer from reward hacking in large-scale reinforcement learning. For general data in the later DeepSeek-R1 model, they did use reward models to capture human preferences.

What does an RLVR training step look like?

An RLVR training step samples several attempts per task, scores each attempt with the verifier, and updates the model so that higher-scoring attempts become more likely. The steps below show one common shape.

1. Sample tasks. Draw a batch of tasks, for example by drawing seeds from the training task families. A seed is a number that fixes every random choice of the task generator, and a task family is a group of tasks that share a structure but differ in their details.
2. Generate attempts. The model attempts each task several times. In an agent environment, each attempt is a full episode with many tool calls.
3. Verify. The verifier scores every attempt. In an environment, it reads the final state after submission.
4. Update. The training algorithm makes higher-scoring behaviour more likely, usually with a penalty that keeps the model close to where it started.
5. Repeat with new tasks, watching both the training reward and the score on held-out tasks.

The update in step 4 depends on the algorithm. Tulu 3 used PPO, which trains a separate value model, also called a critic, alongside the model to estimate how good each attempt was expected to be. DeepSeek-R1 used Group Relative Policy Optimization (GRPO), which drops that estimate and scores each attempt against the other attempts at the same task: its reward minus the group's mean, divided by the group's standard deviation. Both objectives include a Kullback–Leibler (KL) penalty, which keeps the trained model close to a reference model.

With a group method, sampling several attempts per task matters a great deal. If every attempt in a group gets the same reward (all fail or all succeed), every attempt's advantage is zero and the task adds no reward signal; only the KL penalty still acts. Tasks at the edge of the model's ability give the most useful signal, which is why calibrated difficulty matters so much for training environments.

Which domains are classically verifiable?

Maths, code and instruction following are the classic verifiable domains, because each has a cheap, exact check: a known answer, a set of unit tests, or constraints on the form of the output.

Code is a well-studied case. In their evaluation of code-generating models, Chen et al. (2021) used functional correctness, where a sample counts as correct if it passes a set of unit tests, instead of comparing it with a reference solution text.

In all three domains, there is a defined target, the check can be computed, and passing it does not depend on anyone's taste.

What makes a reward verifiable?

A reward is verifiable when its target is defined in advance and its check is computable. It is safe to optimise when, in addition, the check covers everything that matters and the model cannot reach it. A reward that misses one of these conditions can be gamed or can teach the wrong thing.

• The target is defined in advance. You can write down what a correct result is before the model produces anything.
• The check is computable. Code can decide it, with no human and no model in the loop.
• The check is complete for what matters. It tests everything that would make the result wrong, not only the easy parts. A sorting function that passes tests on already-sorted lists has not been verified.
• The check is out of reach. The model cannot see the expected answer or change the code that checks it.

The classic domains meet the first two conditions and only partly the last two: unit tests cover only the cases someone wrote, and instruction checks measure form. A common problem is an incomplete check.

A check that covers part of what matters still produces numbers, and the model learns to satisfy that part. A check within the model's reach is the other frequent failure: models that can edit tests or scoring code may do so, as METR reported in 2025. In code, for example, a model with file access may be able to edit the tests themselves.

The definition used in this guide is strict: no model in the scoring. In practice the line can blur. When filtering data for supervised fine-tuning, not as a reinforcement learning reward, DeepSeek-R1 fed the ground truth and the model's prediction into DeepSeek-V3 and let it judge, which is a model-based check even though a correct answer exists.

How do verifiers differ from learned reward models?

A verifier is code that decides correctness; a reward model is a trained model that predicts how good an output is. They suit different problems and fail in different ways.

Reinforcement learning from human feedback (RLHF) is training a model against a reward model learned from human preference rankings. In the InstructGPT work described by Ouyang et al. (2022), people compare outputs, a reward model is trained to predict which output they would prefer, and the language model is optimised against it. This works for qualities with no exact answer, such as helpfulness.

The weakness of a reward model is validity, not repeatability. A frozen reward model gives the same output the same score, but as the model being trained produces outputs unlike those the reward model learned from, its judgement gets less accurate, and optimisation finds outputs it overrates. Gao et al. (2022) measured this effect: because the reward model is an imperfect proxy, optimising it too far can hurt true performance. An exact check stays valid on any output it can parse, within what it checks.

The two can be combined, as long as the verifier decides correctness. Deterministic verifiers vs. LLM-as-a-judge shows hybrid designs.

How does RLVR extend from answers to workflows?

RLVR extends to workflows when the verifier checks the final state of a world instead of a single answer. The agent works through many tool calls, and the reward is computed from what the world looks like afterwards.

Agent benchmarks already take this step. τ-bench, for example, evaluates an agent by comparing the database state at the end of a conversation with the annotated goal state, and by checking that the agent's replies to the user contain the required information. Confident wording alone cannot pass a task; the database has to end in the right state.

Business work fits this pattern well, because many back-office tasks end in a state that can be checked against written rules.

Three ingredients turn such work into a verifiable reward:

1. Encoded rules that define what is allowed, used by both the task generator and the verifier. Test the encoded rules on their own, against hand-checked cases: a wrong rule shared by the generator and the verifier passes both.
2. A reference to score quality against, where many answers are valid. A reference solution is a known good answer, computed once per seed and frozen so it never changes. For planning tasks, such as rosters, a constraint solver can compute it; Solver-computed reference solutions explains how.
3. Layered scoring: hard gates that must pass, then weighted scores with reasons. A gate is a check that sets the score to zero when it fails: for the whole task when the failure makes the whole result unusable, or for one part when each part is scored on its own. Scoring agent tasks works through an example.

Vencur's worlds follow this design, with no model in the scoring.

How do you turn a workflow into a verifiable task?

Turn a workflow into a verifiable task by writing down the end state, splitting the checks into gates and scores, and deciding what the reference is. The 40-laptop purchase task makes a good worked case.

In this specification, the gates decide whether the work counts at all. An order with an unlisted supplier scores zero, however cheap it is. The scores then rank the orders that pass: a correct order from a more expensive supplier still earns most of the points, with a reason that names the cheaper option.

The reference for the purchase task is simple: the cheapest listed supplier that can deliver, which the generator knows.

How do you write a good verifier for RLVR?

A good verifier is deterministic, complete for what matters, gated, explained, fast, separate from the agent and versioned. Each property protects the reward from a different failure.

Writing the verifier can be harder than writing the task. A useful habit is to write the verifier first and then build tasks that it can score, rather than the other way round.

How do you know RLVR is working?

RLVR is working when scores rise on tasks the model never trained on and a sample of episodes shows real solutions. Held-out tasks catch overfitting; only reading episodes or an independent check catches an exploit of the verifier itself, because held-out tasks are scored by the same verifier.

Watch five signals:

• Held-out score. Keep some seeds and some task families out of training. If the training reward rises but the held-out score does not, the model is overfitting to the training tasks or memorising answers.
• Reliability. Run each held-out task several times. τ-bench introduced pass^k, the chance that all k attempts at a task succeed. A model that solves a task once in five tries is not yet reliable.
• Score breakdown. Look at which gates and score layers improve. Progress concentrated in one easy layer can hide stagnation in the hard ones.
• Length and variety. Track how long the attempts are and how much they differ from each other. A sudden change in either is a reason to read episodes.
• Trajectories. Read a sample of episodes by hand after each run. Some exploits are easy for a human reader to spot in a log even when the aggregate numbers look fine.

What are the limits of RLVR?

RLVR has four main limits: sparse rewards on long tasks, outcome-only signals, qualities that cannot be checked exactly, and verifiers that can be gamed.

Sparse rewards. On a long task, a single pass or fail at the end gives the model little information, especially early in training when it rarely succeeds. Three remedies help: gating and scoring each part of a task on its own, so a mostly correct attempt still earns something; checking important intermediate states; and choosing tasks at a calibrated difficulty. Partial credit after the gates helps in a different way: it ranks attempts that already pass, but it adds nothing while most attempts fail a gate.

Outcome-only signals. A correct final answer can come from flawed reasoning. In maths, Lightman et al. (2023) compared reward models trained with outcome supervision, feedback on the final result, and with process supervision, feedback on each step. Used to pick the best of many sampled solutions, the process-supervised model did significantly better.

That comparison was about choosing among samples, not about reinforcement learning, and the DeepSeek-R1 report lists a process reward model among its unsuccessful attempts at large-scale reinforcement learning. For workflows, checking intermediate states where they matter, such as an approval recorded before an order, is a design choice that brings in some step-level signal.

Coverage. Some qualities cannot be checked exactly: the tone of a message, the clarity of a summary. RLVR does not reach them, and forcing them into a check usually creates a bad one.

Gaming. A verifier with gaps can teach the model to use them. In 2025, METR reported frontier models trying, often successfully, to raise their scores by modifying tests or scoring code.

How do you design verifiable rewards that resist hacking?

Design the reward around the real outcome, keep it out of the agent's reach, and attack it before training. Beyond the seven verifier properties, three practices matter most.

1. Score honesty. Compare the agent's final report with the state, and take points away for false claims. Honesty checks explains how to score final reports.
2. Use fresh seeds and hidden tasks. Generate tasks from seeds, and hold some families back, so memorised answers do not pay.
3. Red-team the environment. Try the obvious exploits yourself, including reading or editing the verifier, and close each gap before training.

Reward hacking in AI agents goes deeper into the defences. For the choice between environments and other training data, see RL environments vs. human-labelled data. To see how Vencur's worlds apply honesty scoring, seeded task families and red-teaming, visit Vencur Environments.

In this series

• Deterministic verifiers vs. LLM-as-a-judge: how to score agent work
• Reward hacking in AI agents: what it is and how to prevent it
• Solver-computed reference solutions: constraint solvers as ground truth
• Scoring agent tasks: gates, weights and partial credit
• Honesty checks: catching agents that claim work they didn't do