Reproducible agent evaluation: seeds, frozen references and episode logs

A reproducible agent evaluation fixes everything on the environment side (seeds, versions, reference solutions and external services) and measures what cannot be fixed, the model's own variation, by running each task several times. It records all of it, so any score can be repeated and audited later.

This guide covers where non-determinism comes from, which parts can be fixed and how, why offline runs matter, how hash-chained logs work and what a report should contain. For the wider question of what to measure, see How to evaluate AI agents.

Where does non-determinism in agent evaluation come from?

Non-determinism in agent evaluation comes from four main sources: the model's sampling, changes in tools and APIs, time, and external services; simulated users, model judges and infrastructure failures add more in some setups. Tools, time and external services can be fixed on the environment side; the model and any changes its provider makes can only be recorded and measured.

Model sampling is the source evaluators cannot remove. Language models sample their outputs, and even settings meant to reduce variation do not guarantee identical results: the OpenAI API documentation, for example, states that its chat completions are non-deterministic by default, and that changes to model configurations on its side can affect determinism. Its responses carry a system_fingerprint field so that such changes can be tracked.

Tool and API drift changes results without anyone noticing. A tool that gains an argument, a policy text that is edited or a model updated behind the same name can each shift scores.

Time enters through clocks. A world that reads the real date behaves differently next month, and a rule check that crosses midnight or a clock change gives different answers if it reads local time.

External services are outside the evaluator's control. A search result, a web page or a third-party API can change between two runs of the same task.

How do seeds and versioned environments make runs repeatable?

Seeds make the tasks repeatable, and versions make the environment repeatable. Together they make two runs of the same task start from the same world.

A seed is a number that fixes every random choice of an environment, both when a task is generated and during the episode. The Gymnasium interface makes this explicit: the seed passed to reset initialises the environment's random number generator. A seed only fixes what draws from it, so every random choice in the world, including noise in documents and the replies of simulated people, must come from generators derived from the task's seed.

Seeds have limits that matter in practice. Python's documentation says that by reusing a seed, the same sequence should be reproducible as long as multiple threads are not running, and that most of the random module's algorithms and seeding functions are subject to change across Python versions. Pin the language and library versions with the environment, and avoid drawing random numbers from several threads at once.

Version everything that can change a score, and record the versions with every episode:

• The environment: the world, its rules and the task generator.
• The tools: their names, arguments and behaviour.
• The verifier: every check and weight.
• The references: the frozen reference solutions, described below.

A change to any of them gets a new version number, and old versions stay runnable, so earlier results can be repeated exactly.

Why freeze reference solutions?

Freeze reference solutions so that every score is measured against the same target, today and in a year. A reference recomputed later can differ, and then scores drift for reasons the agent does not control.

A reference solution is a known good answer to a task, used as the target when scoring quality. For planning tasks it is often computed with a constraint solver, and a limit in seconds depends on how fast the machine is, so a solver stopped by such a limit may not reach the same point everywhere. OR-Tools' CP-SAT solver, for example, offers a limit in deterministic time, a measure of solver effort rather than of seconds, as well as one in seconds. On its own it does not make a parallel search repeatable, so fix the number of search workers as well, and record both settings.

The safe practice is to compute each reference once, with fixed settings, store it with its status and settings, and never recompute it during an evaluation. Solver-computed reference solutions explains what to store and how to check a reference before freezing it.

Why run evaluations offline?

Run evaluations offline because external services are a source of variation the evaluator cannot control. An environment that makes no calls outside its own container gives every run the same world.

Offline operation means three things in practice:

1. No live web or third-party APIs inside the world. Anything the agent needs to look up is part of the generated world.
2. A fixed clock. The world has its own date and time, set by the task, not read from the machine.
3. Pinned dependencies. The container image carries the exact versions of the language, the libraries and the tools.

The model itself may still be called over an API, and then its version and settings are recorded with each episode. Everything else stays inside the container. Gym-style APIs for LLM agents describes an interface for running environments this way.

How do hash-chained episode logs make tampering visible?

A hash-chained episode log is a record of every step of an episode in which each entry's hash covers the previous hash and the entry itself, so changing any entry changes every hash after it. With its final hash stored somewhere safe, the log shows whether it has changed since; it does not show that the entries were true when they were written.

The rule is hash_i = SHA-256(hash_{i−1} + entry_i), starting from a fixed value. SHA-256 is defined in the US federal standard FIPS 180-4, which says such digests are used to detect whether messages have been changed since the digests were generated.

import hashlib, json

def chain_hash(lines, start="0" * 64):
    h = start
    for line in lines:
        h = hashlib.sha256((h + line).encode("utf-8")).hexdigest()
    return h

def verify(log_lines, stored_final_hash):
    """True if the log is the one that was written, given a final hash kept elsewhere."""
    return chain_hash(log_lines) == stored_final_hash

if __name__ == "__main__":
    log = [json.dumps({"step": 1, "tool": "get_invoice", "result": "duplicate line found"}, sort_keys=True),
           json.dumps({"step": 2, "tool": "submit_report", "result": "accepted"}, sort_keys=True)]
    stored = chain_hash(log)                     # written to a separate record when the episode ends
    print(verify(log, stored))                   # True
    tampered = log[:1] + [log[1].replace("accepted", "rejected")]
    print(verify(tampered, stored))              # False

Run as written, the check prints True for the original log and False for the edited one. Two details matter. First, each entry is serialised the same way every time, here with sorted keys, so the same entry always gives the same hash. Second, the final hash is written to a separate record when the episode ends; someone who can rewrite both the log and that record can rebuild the whole chain.

Inside an episode walks through a complete illustrative log with its hash values.

What should a reproducible evaluation report contain?

A reproducible evaluation report contains every setting needed to run the evaluation again and every number needed to judge its result. The table below lists the essentials.

{
  "model": {"name": "example-model", "version": "2026-09-01", "provider_fingerprint": "fp_..."},
  "sampling": {"temperature": 0.0, "max_tokens": 4096},
  "environment": {"world": "invoice-desk", "world_version": "1.4.0", "tools_version": "3.0", "verifier_version": "2.1", "references": "refs-2026-10"},
  "tasks": {"families": ["duplicates", "price-differences"], "seeds": "900001-900200", "held_out": true},
  "runs_per_task": 5,
  "container_image": "sha256:...",
  "log_root_hashes": "stored separately in the run record"
}

A manifest like this turns "we ran the evaluation" into something another team can repeat: the same model version, the same settings, the same world and the same seeds.

Report the uncertainty, not only the average. When two agents run on the same tasks and seeds, compare them task by task: a confidence interval on the per-task difference is usually narrower than two separate intervals, because it removes the differences between tasks. Repeated runs of one task are not independent samples of the evaluation, so compute uncertainty with tasks as the unit; more runs per task reduce the noise within each task, while only more tasks reduce the uncertainty from which tasks were chosen.

How do you check that an evaluation is reproducible?

Check reproducibility by repeating parts of the evaluation and comparing the results. Three useful checks:

1. Replay recorded actions. Take the recorded actions of a few episodes and replay them against fresh instances with the same seeds. The observations, final states and scores must match exactly; if they do not, the environment side is not deterministic. This assumes that simulated people in the world are deterministic code or replay recorded replies.
2. Re-score saved final states. Run the verifier again on saved final states, on a second machine. The scores must be identical, which holds for a verifier written as deterministic code but not for one that asks a model to judge.
3. Repeat a small evaluation. Run a sample of tasks again with the same settings and compare the results with the original, within their confidence intervals. A result well outside those intervals points to a change in the model, a setting or the environment.

The first two checks test the environment and the verifier, which should be exactly repeatable. The third tests the whole evaluation, where only statistical agreement can be expected.

To run reproducible evaluations on operations work, see Vencur Environments.