Realistic mess: why training worlds need noisy documents and conflicting rules

Training worlds need realistic mess because agents learn what their inputs look like: an agent trained only on clean documents and consistent rules can come to expect them, and then fail on real ones. Mess belongs in the inputs, never in the scoring, so the verifier can still check the final state exactly.

This article explains what realistic mess means for operations work, why it matters, how to keep verification exact and how to generate mess from seeds.

What does real-world mess look like?

Real-world mess is everything that makes inputs harder to read than the facts behind them. In operations work, it comes in five common forms, and a sixth difficulty, conflicting rules, often comes with them.

Illustrative example.

A works agreement here means an agreement between an employer and its elected employee representatives. Conflicting rules are not noise in the strict sense: which rule wins is part of the task, written down in the world. They belong in the list because they arrive mixed with the mess, and an agent has to read through one to apply the other.

None of these kinds of mess is exotic. They are what a person on an operations desk deals with every day, usually without noticing, because experience tells them which document is current and which rule wins.

Why does mess matter for agents in real use?

Mess matters because an agent can learn the shape of its training inputs along with the task. If every training document is clean, nothing teaches the agent to check: an agent might read the first date it finds, trust the newest message or apply the most explicit rule, and still score well.

In real use, habits learned on clean inputs break. A date in an unexpected format is misread, an outdated policy is applied, or a manager's request overrides a rule that ranks higher. Each error is small, and in a long task, small errors compound.

Robotics faced a similar problem between simulation and hardware. Tobin et al. (2017) trained an object detector only on simulated images with random, non-realistic textures, and argued that with enough variability in the simulator, the real world may appear to the model as just another variation.

Documents and rules are not images, but the idea may carry over: variety in the training inputs can make real inputs look familiar. One lesson from that result for setting rates, by analogy rather than from the paper itself: match the kinds of mess found in real inputs, but spread the rates over a range wider than the real one, so real inputs fall inside what the agent has seen. The kinds and the range matter more than surface realism.

How do you keep verification exact with messy inputs?

Keep the truth in a structured state and make every document a noisy view of it. A verifier is the code that checks the final state and computes the score; the agent reads the noisy views, the verifier reads the truth.

An RL environment is a software world in which an agent acts through tools while code scores the outcome. A world is a complete simulated workplace packaged as one RL environment, with its data, documents, rules, tasks and verifier.

Illustrative example: in a world built this way, a sick note is generated from a record that says "W-014 is absent on 13 and 14 October". The note may show the dates in any format, with a character error or a missing line, but the record does not change. When the agent submits, the verifier checks the roster against the record and the rules, exactly as it would in a clean world.

Three rules keep the design honest:

1. Every fact can be recovered. Every fact the task needs can be recovered: from the document, from a second source such as a message, or, if it is lost everywhere, by asking for it. The generator records which facts a noisy document lost and where else they can be found.

   If a fact is lost everywhere, the correct end state is to ask for it. The verifier accepts a request only in that case and penalises a request for a fact that was readable, so that always asking does not pay; Reward hacking in AI agents explains why such escape hatches need a check.

2. Noise never changes the rules. A conflict between a works agreement and a manager's request arrives in noisy documents, but which one wins is a rule in the world, written down and checked.

3. The verifier never reads the noisy view. Scoring from a generated document instead of the state would bring the noise into the score.

How do you generate mess from seeds?

Generate mess with the same seeds as the rest of the task, so that every noisy document can be reproduced exactly. A seed is a number that fixes every random choice of the task generator.

import random
from datetime import date

DATE_FORMATS = ["%d/%m/%Y", "%d.%m.%y", "%Y-%m-%d", "%d %B", "%A %d"]
OCR_SWAPS = {"0": "O", "1": "l", "5": "S", "8": "B"}

def noisy_sick_note(seed, staff, first_day, last_day):
    """Render one fact (an absence) as a messy document. The truth stays in the state."""
    fmt_rng, ocr_rng, cut_rng = (random.Random(f"{seed}:{kind}") for kind in ("format", "ocr", "cut"))
    fmt = fmt_rng.choice(DATE_FORMATS)
    text = (f"Medical certificate for {staff}: unfit for work "
            f"from {first_day.strftime(fmt)} to {last_day.strftime(fmt)}.")
    if ocr_rng.random() < 0.3:                  # a scan with character errors
        text = "".join(OCR_SWAPS.get(c, c) if ocr_rng.random() < 0.15 else c for c in text)
    cut = cut_rng.random() < 0.2                # a photo of the page, last third missing
    if cut:
        text = text[: len(text) * 2 // 3]
    truth = {"staff": staff, "absent_from": first_day.isoformat(), "absent_to": last_day.isoformat()}
    present = {"staff_id_intact": staff in text, "dates_present": not cut}  # recorded for the verifier
    return text, truth, present

if __name__ == "__main__":
    for seed in (1, 2, 3):
        text, truth, present = noisy_sick_note(seed, "W-014", date(2026, 10, 13), date(2026, 10, 14))
        print(seed, text, present)
    print(truth)

Run as written, seed 1 gives the dates as "13/10/2026". Seed 2 gives "Tuesday l3 to Wednesday 14", with a scanning error that turns a 1 into an l, and seed 3 gives "l3 October to 14 October". The truth is the same for every seed, and the same seed always gives the same text.

Across seeds 1 to 200, 40 notes lose both dates, and 11 have a damaged staff ID, such as "W-0l4", so a careless agent may not match the note to W-014. The present record only marks whether the staff ID survived exactly and whether the dates were cut off; it does not record scanning damage to the dates, such as seed 2's "l3". A real generator must also record whether each fact can still be recovered, from the note itself or from a second source such as the employee's message, and where. Only then can the verifier tell a justified request for information from an unnecessary one.

A few practices make generated mess useful:

• Use one random generator per kind of noise, each derived from the task's seed, as the sketch does with random.Random(f"{seed}:ocr"). A single shared generator would shift every later choice when a new kind of noise is added.
• Set a rate per kind of noise, and expose the rates as difficulty settings. A task family is a group of tasks that share a structure and are generated from different seeds; with rate settings, a family can start clean and become messier as agents improve.
• Make scan errors look real. A lookup table of character swaps is easy for an agent to learn. A more realistic pipeline renders the document as an image, degrades it with blur, rotation and compression, and runs a real OCR engine over it; that is mainly a cheap way to get the right kinds of error, and comparing OCR output on a few real scans with hand transcriptions gives the real error rates. For agents that read images directly, give them the degraded image itself.
• Avoid tells. A generated marker such as "[page not legible]" announces the problem, which a real photo never does.
• Include distractors, documents that are accurate but irrelevant to the task, so the agent learns to tell what matters.

What does a messy task look like from start to finish?

A messy task looks like ordinary work with traps in it: the facts are all there, but one is written vaguely, one document is stale and two sources disagree. The example below has three kinds of mess, and each one changes the outcome if the agent gets it wrong.

An agent that reads "this Saturday" as the day before the message was sent changes the wrong week. An agent that trusts the manager's approval, or applies last year's agreement, makes the swap and fails. The verifier needs no judgement for any of this: the roster either shows the swap or it does not, and the message to the manager is checked for its structured reason code, not for its wording.

Which kinds of mess should come first?

Start with the mess that is cheapest to generate and still teaches something: inconsistent formats, then contradictory and outdated documents, then conflicting rules. Scans and photos come last, because they need the most work to generate convincingly.

1. Inconsistent formats, typos and abbreviations are cheap: a generator only needs a list of formats or variants per field, such as the date formats in the sketch.
2. Contradictory and outdated documents need a timeline in the state, so the generator knows which version is current and which message came last.
3. Conflicting rules need the rules to carry a ranking, such as a works agreement above a manager's request, so the verifier knows which one wins.
4. Scans and photos need a rendering and OCR pipeline, and a record of which facts survived.

Giving each kind of mess its own difficulty setting lets a task family add them one at a time as agents improve. Per-kind settings also make failures easier to read: if scores drop when one kind of mess is switched on, that kind is what the agent has not learned yet.

How do you know the mess is realistic enough?

The mess is realistic enough when its kinds match real inputs of the workflow, when it changes what agents learn, and when what they learn carries over to real documents. A world full of noise that makes no difference is decoration.

Three tests help:

1. Compare kinds and rates. List the kinds of noise in a sample of generated tasks and in a small sample of real documents from the workflow. The kinds should match, and the generated rates should cover the real ones.
2. Compare training conditions. Train one agent only on clean tasks and another on messy ones, and test both on messy tasks. If they score the same, and neither is near zero or the maximum, the mess adds little. A higher score for the messy-trained agent shows it learned the generator's mess; the next test shows whether that carries over.
3. Test on real or held-out mess. Evaluate on a few real documents, anonymised, or on a kind of noise that was held out of training. Agents that only learned the generator's patterns fail here.

What makes a good RL environment? lists realistic mess as one of ten properties, with a test for each, and Which workflows make good RL environments? shows which back-office domains are worth the effort.

What does messy input mean for reference solutions?

Messy input changes the reference solution only where it removes a fact from the world entirely. The reference is computed from the true state plus the generator's record of which facts survived: where every fact can be recovered, two tasks with the same true facts but different noise share the same reference; where a fact is lost everywhere, the reference end state includes asking for it.

Keeping the reference tied to the true state, not to the documents, is what keeps scores comparable. A lower score then means the agent misread the mess, not that the target moved. Solver-computed reference solutions explains how references are computed and frozen.

To see how realistic worlds are put together for operations work, visit Vencur Environments.