How to evaluate AI agents: benchmarks, environments and their limits

To evaluate an AI agent, measure five things on tasks close to its real work: how often it succeeds, how reliably it repeats that success, what it costs, whether it stays within safety rules, and whether it reports its work honestly. Public benchmarks cover parts of this; a private evaluation with environments covers the rest.

This guide explains what agent evaluation should measure, what six public benchmarks test and how they score, why pass^k matters, where public benchmarks stop and how to build a private evaluation. For the basics of environments, see What is an RL environment?.

What should agent evaluation measure?

Agent evaluation should measure success, reliability, cost, safety and honesty. Success alone hides most of what decides whether an agent can be used.

Success is the headline number of most benchmark papers. Reliability decides whether an agent can be trusted with recurring work. Cost decides whether it is worth running. Safety and honesty decide whether a person can rely on it without checking everything again.

An AI agent is a system built on a language model that completes multi-step tasks by acting through tools, observing the results and deciding what to do next. Because agents sample their actions, every one of these measures varies between runs, and an evaluation has to run each task more than once.

What do public agent benchmarks test, and how do they score?

The six public benchmarks below test code, web, desktop, assistant, policy-following and office work, and most score by checking the end state or an exact answer. They differ mainly in what the agent works on and how the result is checked.

Three patterns stand out. Most of these benchmarks check an end state or an exact answer, not the agent's description of its work, although some also use a language model to grade parts of it: TheAgentCompany, for example, uses model-based evaluation for checkpoints with less structured deliverables. Several of them, such as WebArena, OSWorld and TheAgentCompany, are built as environments the agent acts in. And each covers a slice of work: TheAgentCompany comes closest to office work, with administration, HR and finance tasks among its roles; the others touch business operations only in parts, such as WebArena's e-commerce and content-management sites or τ-bench's retail and airline policies.

The papers' own results show how hard the tasks were when they were published. The best model in the SWE-bench paper, Claude 2, resolved 1.96% of the issues in one retrieval setting; the revised 2023 WebArena paper reported 14.41% for its best agent against 78.24% for humans; OSWorld's best model reached 12.24% against 72.36% for humans; and GAIA reported 15% for GPT-4 with plugins against 92% for human respondents. These figures describe the agents tested when each paper was written, not current performance.

How do you measure cost, safety and honesty?

Measure cost, safety and honesty per task, next to success, and report them separately instead of folding them into one score. A combined score hides which of them changed.

Cost is what one attempt consumes: tokens, tool calls, wall-clock time or money. Report it per task and per success, because an agent that succeeds more often but costs four times as much may still be the worse choice. Cost also reveals loops: an agent that repeats the same tool call until a step limit cuts it off shows up as an expensive failure.

Safety is whether the agent stays within the rules on the way to its result. Count rule breaks, forbidden actions and actions that cannot be undone, such as a payment made or a message sent to the wrong person, even in episodes that end in success. A world that records every action makes this possible; a benchmark that only checks the final answer cannot see it.

Honesty is whether the agent's final report matches what it did. Compare each claim in a structured report with the final state, and count the episodes with at least one false claim. A false-claim rate of a few per cent matters for work that people act on without checking. Honesty checks explains how to score reports.

Why does reliability need its own measure?

Reliability needs its own measure because an agent that succeeds on a task once may fail on the same task the next time. Average success rates hide that difference.

pass@k is the chance that at least one of k attempts at a task succeeds. pass^k, introduced with τ-bench, is the chance that all k attempts succeed. The τ-bench authors reported that even state-of-the-art function-calling agents succeeded on fewer than 50% of tasks, and that pass^8 in the retail domain was below 25%.

The two measures answer different questions:

• pass@k suits work where results can be checked before they take effect, so a team can run several attempts and keep one that passes the check.
• pass^k suits recurring work whose results are not checked one by one, where every failed attempt becomes a real error.

The arithmetic shows how fast reliability falls on a single task: if an agent succeeds on a task 90% of the time and attempts are independent, pass^8 for that task is 0.9^8, about 43%. Across a benchmark, pass^k is the average of these per-task values, so it depends on how success is spread. An average success rate of 90% can mean pass^8 of about 43%, if every task sits at 90%, or of 90%, if 90% of tasks always succeed and the rest never do. Choose k to match how the agent will be used, for example the number of runs between two human checks.

Both measures are estimated from n runs per task, with n at least k. If c of the n runs of a task succeed, the τ-bench authors give unbiased estimates of pass^k as C(c, k) / C(n, k) and of pass@k as 1 − C(n − c, k) / C(n, k), each averaged over tasks, where C is the binomial coefficient; the pass@k estimator is the one Chen et al. (2021) introduced for code generation.

How many tasks and runs does an evaluation need?

An evaluation needs enough tasks and runs that the difference you care about is larger than the uncertainty of the measurement. A success rate measured on a small sample can move by many points from chance alone.

The arithmetic for a success rate is simple. With n independent tasks and a true success rate near 50%, the standard error is about 0.5 divided by the square root of n, and a 95% confidence interval is roughly two standard errors either side:

Two agents measured on two independent samples of 100 tasks each, both near 50% success, have a standard error of the difference of about 7 points, so a gap of up to about 14 points can be chance. Repeated runs of the same tasks reduce the noise within each task, and more tasks reduce the variation that comes from which tasks were chosen. Report the number of tasks, the runs per task and a confidence interval, for example one bootstrapped over tasks. Comparing two agents on the same tasks, task by task, needs fewer tasks than comparing two independent samples, because it removes the differences between tasks.

What are the limits of public benchmarks?

Public benchmarks are limited by contamination, saturation, narrow domains and, where they use models as judges, the reliability of those judges. Each limit makes a public score say less about how an agent will do on your work.

Contamination

Benchmark contamination is the presence of evaluation data in a model's training data. A survey by Xu et al. (2024) describes it as language models inadvertently incorporating evaluation benchmark information from their training data, which makes results unreliable. Public tasks are exposed by design, so every public benchmark carries the risk.

Some benchmarks defend against it. GAIA's authors released their questions but retained the answers to 300 of them to power a leaderboard, and designed answers that are unlikely to be found in plain text in training data. Benchmark contamination covers detection and defences.

Saturation

Saturation is the state of a benchmark whose top scores approach the maximum, so it stops separating good systems from better ones. Near the top, small differences in score mean little, and whatever errors the benchmark itself contains, such as weak tests or underspecified tasks, weigh more in what is left to measure. Checks in code can also be wrong, so audit a sample of passes and failures by hand, whatever the benchmark.

Narrow domains

Each benchmark covers a few kinds of work, and the kinds most often covered are code, the web and general computer use. An agent that does well on them may still fail at work with dense business rules, long chains of approvals or messy documents. Why AI agents still struggle with back-office work explains which frictions those benchmarks do not reach.

Judge reliability

Some evaluations use a language model as a judge, especially for open-ended answers. Zheng et al. (2023) examined such judges and documented position and verbosity biases, and also found that strong judges agreed with human experts about as often as the experts agreed with each other. A judge is useful for open-ended quality if it is calibrated against human labels, but a score that depends on it carries its biases. Deterministic verifiers vs. LLM-as-a-judge explains when each fits.

Which public benchmark fits which agent?

The benchmark that fits is the one whose tasks are closest to the agent's real work and whose scoring checks what matters for that work. Use it as a first filter, then evaluate on your own tasks.

In every row, the right-hand column is the gap a private evaluation closes. A good public score is evidence that an agent can do a kind of work; it is not evidence that it can do yours.

How do you build a private evaluation with environments?

Build a private evaluation by running the agent in an RL environment for the work it will actually do, on task families that were never used for training, with a deterministic verifier and logs of every episode. Such an evaluation answers the question a public score cannot: how this agent does on this work.

A private evaluation has six parts:

1. Task families from the target work. A task family is a group of tasks that share a structure but differ in their details. Choose families that match the work, including the messy and rule-heavy cases.
2. Fresh, held-out seeds. A seed is a number that fixes every random choice when a task is generated. Unseen seeds rule out memorised instances; held-out families also test generalisation beyond the structures seen in training. Neither helps if the generator, the references or the verifier have leaked.
3. A deterministic verifier. Code checks the final state of each episode against the rules and a reference, so the same end state always gets the same score.
4. Frozen references. Where the quality of the agent's result is scored, such as the wage cost of a roster it produces, a reference solution computed once per seed and frozen gives every score a fixed target.
5. Repeated runs. Run each task several times with fixed sampling settings, and report success, pass^k and the spread.
6. Episode logs. Record every tool call, result, report and score, so any result can be audited later.

The same environment can serve for training and evaluation, as long as the evaluation families and seeds are kept apart. Reproducible agent evaluation explains how to make every score repeatable.

How do public benchmarks and private evaluations work together?

Public benchmarks and private evaluations work best together: public scores compare agents on common ground, and private evaluations decide whether an agent is fit for a specific job. Neither replaces the other.

A practical sequence has three steps:

1. Shortlist with public benchmarks. Use the benchmarks closest to the work to narrow a long list of agents or models to a few candidates.
2. Decide with a private evaluation. Run the candidates on held-out tasks from the target work, with repeated runs, and compare success, reliability, cost, safety and honesty.
3. Re-check over time. Models, prompts and tools change. Repeat the private evaluation on fresh seeds whenever something changes, and keep the results comparable by keeping the task families and the verifier version fixed.

The order matters for cost. Public benchmarks are cheap to read; private evaluations cost compute and engineering, so they are worth spending on the few candidates that remain.

What should an evaluation report contain?

An evaluation report should contain the results for every dimension, per task family, together with everything needed to reproduce them. A single average hides where an agent fails.

Report failures as concretely as successes. The reason for every lost point, computed by the verifier, tells a reader what the agent got wrong; a total alone does not.

What belongs on an evaluation checklist?

An evaluation checklist asks about relevance, freshness, exact checks, reliability, uncertainty, cost, safety and honesty, and reproducibility. Use it for your own evaluations and for any result someone else reports.

• Relevance: do the tasks match the work the agent will do, including messy and rule-heavy cases?
• Freshness: were the tasks kept out of training data, and are they held-out seeds or families?
• Exact checks: is the result checked in code from the end state, or by a judge?
• Reliability: was each task run several times, and is pass^k reported?
• Uncertainty: is a confidence interval reported, and is the difference between agents larger than it?
• Cost: is the cost per task reported?
• Safety and honesty: are rule breaks and false claims counted separately from success?
• Reproducibility: are the model, the scaffold, the tools, the step limits, the seeds, the sampling settings and the logs recorded?

An evaluation that answers yes to all eight says something about how an agent will do in real use. One that answers no to freshness or exact checks says less than its numbers suggest.

To build a private evaluation for operations work, see Vencur Environments.

In this series

• Reproducible agent evaluation: seeds, frozen references and episode logs
• Benchmark contamination: why private, licensed environments matter