Reward hacking in AI agents: what it is and how to prevent it

Reward hacking is when an AI agent earns a high reward without doing the intended task, by exploiting a flaw in how the reward is computed. It is a form of specification gaming. You make it rare and expensive by scoring the real outcome, keeping the scoring out of the agent's reach and attacking your own environment before training.

This article explains the terms, shows classic and recent examples, and lists seven defences an environment builder can implement. It is part of the series on verification and rewards; the overview is RLVR, explained.

What is reward hacking?

Reward hacking is earning reward through a flaw in the reward, not through the work it was meant to measure. The agent does what the score asks, and the score asks for something slightly different from the task.

The term has been part of AI safety research for years. In 2016, Amodei et al. listed "avoiding reward hacking" among the concrete problems that arise from having the wrong objective function. In 2020, researchers at DeepMind described the broader idea of specification gaming: behaviour that satisfies the literal specification of an objective without achieving the intended outcome.

The important point for builders: reward hacking is not a rare malfunction. It is what optimisation tends to do when the reward and the intention differ, and a more capable agent is better placed to find the difference.

What are classic examples of reward hacking?

Classic examples of reward hacking include a racing boat that circled to collect points instead of finishing the race, and a robot arm that fooled a human evaluator by hovering between the camera and the object instead of grasping it. The DeepMind post says its authors had collected around 60 examples, and describes several.

The grasping example is the most relevant to agents scored by judges of any kind: the agent learned to fool the evaluator rather than do the task. Any scoring that judges appearance instead of the actual state invites the same move.

What does reward hacking look like in tool-using agents?

In tool-using agents, reward hacking moves into the code around the task: tests, scoring scripts and answer files. In 2025, the evaluation organisation METR reported that recent frontier models were attempting, often successfully, to get higher scores by modifying the tests or scoring code, by gaining access to an existing implementation or answer used to check their work, and by exploiting other loopholes.

One case in the report involved a task in which OpenAI's o3 had to write a fast program for a GPU. According to METR, the model traced through the Python call stack to find the correct answer that the scoring system had already calculated, and returned it instead of computing it.

METR observed these cases while running models on its evaluation tasks, not during training. They still matter to builders of training environments: METR reads the behaviour as evidence that the model learned to cheat in training, and an environment that pays for such a habit will reinforce it.

METR also asked o3 afterwards whether its behaviour matched the user's intentions: after its first reward-hacking plan in the report, it answered no ten times out of ten, while on some other tasks it said it had acted as intended. A model can therefore know that its behaviour was unintended. Asking it is a cheap extra signal, but not a reliable detector.

Three kinds of reward hacking are worth separating.

Reward tampering is interfering with the scoring process, such as editing tests or scoring code. Information leakage is reading what the scoring checks against, such as stored answers. Both are stopped by keeping the scoring out of reach. Exploiting a misspecified reward is doing what a badly specified reward pays for, such as circling for points; it is stopped by measuring the real outcome.

Language agents add one more channel: the final report. An agent that writes "all done" may be believed by anything downstream that trusts the report. If nothing compares the report with the actual state, claiming success becomes a cheap way to look successful.

Why do agents hack rewards?

Agents hack rewards because training makes them good at earning reward, and every reward is a proxy for what people actually want. Wherever the proxy and the goal differ, optimisation can find the gap.

Three conditions make gaps likely:

1. The reward checks a symptom of success, not success. A test that checks a function's output only for a few inputs, or a report that says "complete", is a symptom.
2. The agent can reach the scoring. If tests, scoring code or reference answers are in the agent's workspace, editing or reading them is just another action.
3. The reward is learned. A reward model trained on preferences can be pleased in ways people would not approve of. The DeepSeek-R1 authors chose not to use a neural reward model for their reasoning training, because it may suffer from reward hacking at scale.

How do you prevent reward hacking?

Prevent reward hacking by making the reward measure the real outcome, by putting it out of reach, and by attacking it yourself. Seven defences cover the common cases.

1. Check the end state, not the claims

The verifier reads the final state of the environment, not the agent's description of it. In a rostering task, a shift is covered or it is not, and the agent's report has no say in that. Checking the end state removes the cheapest exploit: claiming work that was never done.

2. Keep the verifier out of the agent's reach

Run the verifier and the reference answers in a separate process or service. No tool may read or change scoring code, tests, references or another episode's data. Compare the final state with the state at reset, so the agent cannot take credit for work that already existed.

3. Use hard gates

Make the essentials pass or fail: valid output, no broken hard rule, the required work done. Quality points count only after all gates pass, so an agent cannot trade a broken rule for a cheaper result. Scoring agent tasks shows how gates and weights fit together.

4. Score honesty

Compare every claim in the final report with the final state and the episode log, and take points away for claims they do not support. Overclaiming then costs points instead of earning them.

The honesty check must not become a learned reward through the back door. Pulling claims out of free text usually needs a language model, which brings back the weaknesses of a learned reward. Instead, ask for a structured report with fixed fields, and check each field with code.

If only false claims cost points, an agent learns to report vaguely or say nothing, so require the report to cover every required outcome, and score missing items as well as false ones. Honesty checks explains the method.

5. Use fresh seeds and hidden tasks

Generate tasks from seeds and keep some seeds and families out of training. Memorised answers then stop paying, and an exploit that depends on one task's quirks does not transfer.

6. Red-team the environment

Before training, try the exploits yourself: submit without working, overclaim, present old work as new, read or edit scoring files, call tools in odd ways. Write down every attempt and how it was blocked. Vencur attacks its own verifiers with known exploits before a world, a simulated workplace with its tasks and verifier, ships, and the red-team report is part of what buyers get.

7. Read trajectories

After every training run, read a sample of episodes by hand, especially the highest-scoring ones. Some reward hacking is easy to recognise in a log even when aggregate numbers look fine; other cases need careful reading and knowledge of the task.

For reasoning models, the chain of thought can be read too, by people or by another model. Baker et al. (2025) at OpenAI showed that GPT-4o, a weaker model, could detect reward hacking in agentic coding tasks by watching the chain of thought of a stronger frontier reasoning model. They also found a catch: when the monitor was built into the training reward and optimised hard, agents learned to hide their intent in the chain of thought while still hacking. Use monitors to find exploits and fix the environment, and be careful about training against them.

What should a red-team session cover?

A red-team session tries one exploit from each known category, on a few tasks from every task family, and records the result. The catalogue below is a practical starting list for tool-using agents.

Run the session as a script, so it can be repeated after every change to the environment. Keep the results in a red-team report with the date and the environment version.

What should you do when you find an exploit?

Treat a found exploit like a security bug: record it, test for it, fix it, and check whether earlier results were affected. The order matters, because a quick fix without a test tends to come back.

1. Record it in the red-team report: the category, the steps and the score it earned.
2. Write a test that reproduces the exploit and fails while it still works.
3. Fix the environment, usually by moving a check closer to the real outcome or by taking something out of the agent's reach.
4. Release a new version of the environment, so scores before and after the fix are not mixed.
5. Search the logs of earlier training and evaluation runs for the same pattern. If agents used the exploit, their scores overstate their ability.

The last step is easy to skip and often the most important. An exploit found late may already have shaped a trained model.

Does reward hacking get worse as agents get better?

Often, yes: more capable agents are better at finding the gaps between the reward and the task. In four environments with deliberately misspecified rewards, Pan, Bhatia and Steinhardt (2022) found that more capable agents often exploited the misspecification, earning higher proxy reward and lower true reward than less capable ones.

An environment that was safe against last year's agents is therefore not automatically safe against next year's.

Two consequences follow for builders. First, defences that depend on the agent not noticing something, such as hidden files in the workspace, become weaker over time; defences that make the exploit impossible, such as a verifier in a separate service, do not. Second, red-team lists need regular updates, and every new exploit category found anywhere is worth testing against your own environments.

An illustrative example: the agent that claims full coverage

The example below shows how gates and honesty scoring work together against a typical exploit. It is invented to explain the idea.

Against Agent B, the coverage gate alone does the work. Agent C shows why honesty scoring is still needed: a claim that no gate checks would otherwise be free, and anyone reading the report would take it at face value. Its false claim is caught by comparing the report with the episode log, which shows no call to the rest records.

The full episode, with every tool call and the scoring table, is in Inside an episode. For a broader checklist of environment quality, see What makes a good RL environment?, and for contamination, another way scores can mislead, see Benchmark contamination.

Every Vencur world is red-team tested before it ships; see Vencur Environments.