Two documents, one policy, four disagreements

This morning I re-ran the PolicyTrace demo for the first time in weeks, and it failed. Groq had retired the Llama model it called, so every request came back model_not_found. I switched to openai/gpt-oss-120b. Then the Schedule extraction came back cut off halfway, because gpt-oss is a reasoning model: it spends part of its token budget thinking before it answers. With a bigger budget and reasoning effort set to low, the run finished. It produced 59 fields, 49 of them traced to a line in the PDF, plus four conflicts between the Schedule and the Certificate.

The conflicts are the interesting part, so this post is about them.

Why two documents can disagree

A UK motor policy usually arrives as a pack: the Schedule, the Certificate of Motor Insurance, a Statement of Fact and the policy booklet. They overlap, but each is written for a different job. The Schedule is the commercial summary, covering the vehicle, drivers, excesses and premium. The Certificate is the legal document you would show the police. That's why it's the one I trust for class of use and for whether you can drive other cars.

PolicyTrace extracts each document separately, then merges the results. The merge doesn't ask the model who is right. It's plain Python in arbiter.py, and the rule fits in one sentence: the Schedule wins, except for class of use and driving other cars, where the Certificate wins. Whenever both documents have a value and the values differ, the arbiter records a conflict with the field, both values and the winner.

I'd rather keep that rule in code than in a prompt. I can test it, a domain owner can read it, and changing it doesn't mean re-tuning extraction.

What the four conflicts were

1. Insurer name. "Northbridge Mutual Motor  Insurance Ltd" against "Northbridge Mutual Motor Insurance Ltd". The difference is a double space left behind by PDF text extraction, so this isn't a real disagreement.
2. Class of use. The same story: a double space inside a long sentence. The Certificate won, and it happened to be the clean version.
3. Main driver, policyholder. The Schedule says yes, the Certificate says no.
4. Main driver, a named driver. The reverse.

So half the conflicts were noise. The comparison lowercased and trimmed both values but didn't collapse the spaces inside them. That's a one-line fix, and it's now in. Left alone, it would have taught reviewers to skim past the conflict panel, which is the worst thing that can happen to a feature whose only job is to get attention. The panel didn't help either: it explained every conflict as "wording differ", even when the conflict was a true/false flag. It now shows both values side by side.

The bug the conflict panel caught

The two main-driver conflicts are more interesting. Both say the Schedule won. For the named driver the Schedule says not the main driver, so the Golden Record should hold is_main_driver: false. It held true. The record had two main drivers.

Here was the line responsible:

is_main_driver=sd.is_main_driver or (cd.is_main_driver if cd else False),

Because of the or, a True from either document wins, whatever the conflict entry says. I wrote it to fill a gap when the Schedule didn't mention the main driver. But the schema defaults is_main_driver to False, so the code can't tell "the Schedule said no" apart from "the Schedule said nothing". The proper fix has two parts:

# schema.py
is_main_driver: Optional[bool] = None

# arbiter.py: same rule as every other field, a missing value is None
is_main_driver=_first(sd.is_main_driver, cd.is_main_driver if cd else None),

Both changes are now in the repo, each with a regression test that fails on the old code.

What I like about this bug is how it surfaced. It was visible only because the arbiter recorded the conflict. If the merge had quietly picked a value, I'd have a record with two main drivers and nothing pointing at it. A conflict entry that disagrees with the record is exactly the kind of thing a reviewer, or a test, can catch.

What I'd add before trusting this with real policies

• A test that the record matches the conflict ledger. For every conflict, assert that the Golden Record holds the winner's value. That would have caught this the day I wrote it. It's in the test suite now.
• Normalise text before comparing. Collapse whitespace and case, and treat "£200" and "GBP 200.00" as the same value.
• Business rules the arbiter doesn't know about. There should be exactly one main driver, and a provisional licence holder shouldn't normally be one.
• A stricter idea of "located". On that run, 49 of 59 fields had a highlight, but many of those highlights sit on the label ("Policy number") rather than on the value next to it. A highlight on the right line isn't proof that the value is right.

None of this is exotic. It's the unglamorous part of document AI: the model got the extraction mostly right, and the mistakes worth talking about were in my own merge code.

Try it yourself

The live demo loads a synthetic four-PDF pack in one click. With both fixes deployed, the same pack now comes back with one main driver and no whitespace conflicts. A full run takes about three minutes on Groq's free tier, which mostly means waiting on rate limits. The code, including arbiter.py, is on GitHub. If you've solved the "which document wins" problem differently in your own pipelines, I'd like to hear how.

Update, later on 17 October. A closer look showed the two main-driver conflicts above weren't real disagreements. The Certificate in the demo pack never says who the main driver is, and names were masked separately in each document, so the arbiter's fuzzy matching paired drivers with the wrong person. Both are fixed: the masker now gives a person the same token across the whole pack, and masked names must match exactly. The same pack now comes back with no conflicts at all. The is_main_driver fix described above still stands. And the highlights on labels I mention at the end turned out to cluster in the driver table; I wrote that up and fixed it too.