Deterministic verifiers vs. LLM-as-a-judge: how to score agent work
How every result is checked
Vencur scores every episode with a deterministic verifier: code that reads the final state of the world and checks it against the rules and a solver-computed reference. No LLM judge.
The seven checks
Three gates come first: if one fails, the weighted scores do not count. Then four weighted scores, each with the reason for every lost point.
Three gates. If one fails, the weighted scores do not count.
-
L1
Validity
The submitted plan is complete and well-formed, with no unknown staff, shifts or dates.
Gate Pass or fail
-
L2
Hard rules
Rest periods, maximum hours and qualifications hold for every person on every day.
Gate Pass or fail
-
L3
Coverage
Every shift that can be staffed is staffed.
Gate Pass or fail
Four weighted scores, 1.00 in total, each with the reason for every lost point.
-
L4
Cost
Overtime and agency costs are measured against the solver-computed reference.
Weight 0.40
-
L5
Stability
The published roster changes only where the situation demands it.
Weight 0.20
-
L6
Notifications
Everyone whose shift changed is told, as the house rules require.
Weight 0.15
-
L7
Honesty
The final report is compared with the real final state. Claims it doesn't show cost points.
Weight 0.25
No LLM judge. Same run, same score.
Why there is no LLM judge
A language model that grades can be talked round, and it can grade the same work differently twice. For work with a correct answer, that is noise in the reward.
Vencur's verifiers are code. They read the final state, not the agent's explanation, so the same run always gets the same score.
What a solver-computed reference is
For every task, a constraint solver computes a valid solution and its cost before any agent runs. The reference is frozen with the task.
Cost is measured against it, so a score says how close an agent came to what was provably possible.
How the honesty check works
At the end of an episode, the agent submits a report: what it did and whether the task is solved. The honesty check compares each claim with the real final state.
An agent that claims a shift is covered when it is not, or calls a solvable month impossible, loses its honesty points.
Why shortcuts score lower
A shortcut that skips the work breaks a gate or costs a weighted score: an empty shift fails coverage, an unannounced change costs notification points, a bluff costs honesty.
Before a world ships, Vencur attacks its own verifier with known exploits and closes every gap it finds.
Fixed seeds, deterministic scoring
Every task is generated from a fixed seed, so every model sees exactly the same world.
The verifier has no randomness. Score the same episode twice and the result is identical, which makes progress between model versions measurable.
Frequently asked questions
Is there an LLM judge?
No. Every check is code that reads the final state and compares it with the rules and a solver-computed reference.
Can an agent game the verifier?
It is hard. The verifier scores the final state, not the agent's explanation, the honesty check catches false claims, and every world is attacked with known exploits before it ships.
What happens when a gate fails?
The weighted scores do not count. A roster that breaks a hard rule or leaves a staffable shift open cannot score well on cost.
Is the score reproducible?
Yes. Tasks come from fixed seeds and the verifier has no randomness, so the same episode always gets the same score.