Part of the PolicyTrace series
Evaluation System note
I swapped the AI model and every test still passed. That was the problem.
Unit tests protect the code around a model. They can't tell you whether a new model still gets the answers right, and that's the change most likely to break things.
• When Groq retired the models PolicyTrace was built on, I swapped in new ones. Every test passed before and after, because none of them called a model.
• Unit tests protect the code around the model: the merge, the highlights, the masking. They can't tell you whether a new model still extracts the right excess.
• The smallest fix I'd trust is one script that runs the real pipeline on a pack with known answers, and is re-run whenever a prompt, model or schema changes.
PolicyTrace reads a car insurance policy pack and returns one checked record. It was built on Llama models served by Groq. When Groq retired them, I switched the extraction to openai/gpt-oss-120b and the document classifier to openai/gpt-oss-20b. The tests passed before the change and after it, in under a second. They would also have passed if the new model had got every premium wrong.
So how did I decide the new model was fine? I ran the demo pack and read the output. That's an honest answer, and it isn't good enough. This post is about what PolicyTrace tests, what it doesn't, and the smallest evaluation I think it needs.
What the tests checked
At the time of the swap, every test was in tests/test_arbiter.py, and they're the right kind of tests for that code. The arbiter is the plain-Python step that merges the Schedule and the Certificate into one record. Its tests build two small records by hand, merge them and check the result. Today they check that:
- the Schedule's vehicle, premium and risk data are kept;
- the Certificate wins on class of use;
- a policy number that differs produces a conflict entry with both values and the winner;
- "ALICE J SMITH" on one document and "ALICE SMITH" on the other are matched as one driver;
- two values that differ only by whitespace are not a conflict;
- for every conflict, the merged record holds the winner's value.
The last two were added after the first live run on the new model, written up in Two documents, one policy, four disagreements. That's the pattern I want to keep: when a run shows a problem, the fix lands with a test that fails on the old code.
What wasn't
Everything before and after the arbiter was checked by eye. Nothing tested that Presidio masked the policyholder's name, that the classifier labelled the four demo PDFs correctly, that the extraction got the right excesses, or that a highlight landed on the value. The same week, a fresh run showed driver fields highlighted on headings and drivers paired with the wrong person. Both were exactly the kind of bug an automated check would flag on its first run.
The closest thing to an evaluation set was manifest.json in the demo pack folder. Next to the file list it has three expected values: the policy number NBM-DEMO-0427, the registration ZX24 DEM and the insurer. That's a start, and it's three fields out of about sixty.
What's tested now
I fixed both bugs and added tests for each. tests/test_masking.py checks that one person gets the same masked token in every document of a pack. tests/test_provenance.py rebuilds the demo's driver table and checks where each field lands. The suite is now 32 tests: 24 for the arbiter, 5 for provenance, 2 for masking and 1 that keeps debug output off by default. Every new one fails on the old code.
None of them calls a model. So the model swap problem is still open.
The smallest evaluation I'd trust
I don't want an evaluation framework. I want one script that runs the real pipeline on the demo pack and compares the result with a file of answers I'm sure of. It needs four things.
An expected record. I generated the demo pack myself with a script in the repo, generate_synthetic_policy_pack.py, so every true value is already in that script's POLICY dictionary. The expected record can be built from it instead of typed out.
Field-aware comparison. Policy number, registration and yes/no fields should match exactly after trimming. Dates need a normalised comparison, because the record holds ISO dates and the PDF holds "15/04/2026". Money needs one too, so that "GBP 395.00" and 395.0 count as the same. And a missing field should be reported apart from a wrong one, because they have different fixes.
The conflicts I expect. I know which disagreements between the Schedule and the Certificate this pack should produce: none. The script should check exactly that.
A check on the evidence. For each located field, is the highlight on the right document, and does the matched text contain the value? Runs report "46 of 59 fields located" (18 Oct). I'd like them to report how many highlights actually point at the value.
A run costs a few minutes of Groq's free tier, which is too slow for every commit. I'd run it by hand whenever I change a prompt in config/prompts.yaml, the model names in config/settings.yaml, the schema or the matcher. Those are the changes that can break extraction without breaking a single unit test, and a model retirement is exactly that kind of change.
Limits
Even with all of that, the result is one synthetic policy. It tells me whether a change made things worse on a pack I know well. It doesn't tell me how PolicyTrace would do on a real insurer's layout, a scanned certificate, or drivers listed in a different order on each document. The generator script is the cheapest way to get more packs: vary the drivers, excesses and layout, and keep the expected record in step. Testing whether a model-based judge can do this marking for you is a separate experiment.
The decision behind it
- Decision: fast unit tests for every piece of plain code, and no model calls in the test suite, so it runs in seconds on every change.
- Alternative rejected: mocking the model's answers in unit tests. That tests my mock, not the model.
- What it cost: the parts that call a model have no automated check at all until the evaluation script exists. A model change is checked by eye.
- When to revisit: now. The evaluation script is the next piece of work on PolicyTrace.
Code: the 32 tests and the demo pack on GitHub
Related: My AI judges agreed with themselves. Not with each other. · The right words. The wrong driver.
Free: the AI Workflow Readiness Checklist, which asks how you'll know a model change made things worse
When you change the model behind a feature, what tells you it got worse?
First published 17 Oct 2026. Updated 25 Oct 2026: new title; the later test additions are part of the story rather than an addendum; new diagram.
Get the next note by email
One short email when something ships or breaks. Free.