How to evaluate AI agents: benchmarks, environments and their limits
Where frontier agents fail
Vencur runs frontier models blind on the hard tasks of OpsDesk-Care, its first live world, and scores every run with the deterministic verifier.
OpsDesk-Care 2.4.13, results updated 23 October 2026.
How the runs work
Results
Hard and expert
13 tasks13 tasks across 8 task families, up to 132 staff and 360 shifts per month. Two swap-request tasks are being revised and are not counted.
-
Claude Opus 5.5 by Anthropic
Fully solved
0 of 13 (0%)
Mean score
0.860
-
Claude Sonnet 5.5 by Anthropic
Fully solved
0 of 13 (0%)
Mean score
0.460
Tiles show a count, not specific tasks.
| Model | Tasks run | Rules kept & honest report | Fully solved | Staffable month called impossible | Mean score |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 13 | 12 of 13 | 0 of 13 | 0 | 0.86 |
| Claude Sonnet 5.5 | 13 | 7 of 13 | 0 of 13 | 5 | 0.46 |
Baseline, October 2026
One run per task on 30 tasks from the DataVendor packs (20 Hard Eval, 10 Honesty), tool interface only, scored by the deterministic verifier.
| Claude Haiku 5.5 (no extended thinking) | Claude Sonnet 5.5 (extended thinking) | |
|---|---|---|
| Fully solved, Hard Eval | 1 of 20 | 1 of 20 |
| Fully solved, Honesty | 3 of 10 | 0 of 10 |
| Feasible month reported as impossible | 7 of 20 | 14 of 20 |
| Impossible month reported as feasible | 3 of 10 | 0 of 10 |
Small sample with one run per task and different thinking settings: the numbers show difficulty, not a model ranking.
What the failures look like
Three kinds of failure from the hard tier, each caught by a different check.
-
Illustrative example. Coverage
A staffable shift left open
The agent finishes with a shift still empty, although a qualified person was free to take it. The coverage gate fails, so the weighted scores do not count.
-
Illustrative example. Honesty
A solvable month reported as impossible
The agent reports that the month cannot be staffed within the rules. The solver found a legal roster for it, so the honesty check marks the report as false and the honesty points are lost.
-
Illustrative example. Cost
A legal roster that costs more than the reference
Every rule holds and every shift is covered, but the roster uses more overtime or agency hours than it needs. The gates pass; the cost score shows the distance to the solver-computed reference.
Frequently asked questions
Which models were tested?
Claude Opus and Claude Sonnet, by Anthropic, on the hard tier; Claude Opus also on the entry tier. Runs with further models are in progress.
What counts as fully solved?
Every gate passed, the honesty check passed and a score of at least 0.999. A run that is almost right is not counted as solved.
Can we run our own agent on these tasks?
Yes. An evaluation pack gives you 25 tasks in your format, scored by the same deterministic verifier.
Are the tasks public?
No. The tasks are not published, so they cannot leak into training data. They are licensed as evaluation packs or for training.
Test your agent on the same tasks
An evaluation pack gives you 25 tasks in your format, scored by the same verifier and invoiced only after you accept it.