Systems / PolicyTrace
PolicyTracePolicyTrace
Reads a UK motor policy pack and gives a claims handler one checked record, with every value traced to the line in the PDF it came from.
The problem
Before a motor claim can move, a handler needs about a dozen facts from the policy pack. Was cover in force on the day? Was the driver on the policy? Was the car being used the way the policy allows? Which excess applies? The pack is four or five PDFs, and the answers are spread across them. Finding them is slow, and missing one costs more.
PolicyTrace does the finding. The handler does the checking, with the source line one click away. I wrote about why I built it separately.
How it works
The model is one step out of six, and each step is a file you can read.
- Parse. Docling turns each PDF into text and keeps every line's position on the page (
agents.py). - Mask. Presidio swaps names, phone numbers and most place names for tokens, with the same token for a person in every document (
privacy.py). - Classify. A small model and a keyword count decide whether each PDF is the Schedule, the Certificate, the Statement of Fact or the booklet.
- Extract. gpt-oss-120b fills a typed form for each document and quotes the phrase it read (
schema.py). - Merge. A written rule decides which document wins and records every disagreement (
arbiter.py). - Trace. Each value is matched back to its line in the PDF, so the reviewer can click straight to it (
provenance.py).
The note What happens to a PDF inside PolicyTrace follows one upload through the code.
Two rules that make it reviewable
Which document wins is code, not a prompt. The Schedule wins, except for class of use and driving other cars, where the Certificate wins, because the Certificate is the legal document. Every disagreement is kept with both values, so the reviewer sees the argument as well as the verdict.
# arbiter.py
class_of_use = _check_conflict(conflicts,
"cover_and_excesses.class_of_use",
sc.class_of_use, cc.class_of_use,
"certificate")The model never gives coordinates. It returns a value and the exact phrase it read. Docling owns the page geometry, and a matcher joins the two, so a highlight always points at real text on the page.
What works, and what doesn't yet
Works today
- One typed record from a four-PDF pack, with a written rule for which document wins.
- Every disagreement between documents is listed with both values.
- Most values highlight on their own line in the PDF, driver rows included.
- Names are masked before the model sees them, with the same token across documents.
- 32 automated tests cover the merge, the matching, the masking and the settings.
Not yet
- Exclusion clauses in the policy booklet aren't extracted.
- Postcodes, street lines and dates of birth reach the model unmasked.
- Names stay masked in the record, so a handler checks them against the PDF.
- There's no sign-in, and review actions keep no history.
- A run takes two to three minutes inside one request.
- It has been evaluated on one synthetic pack, not on real policies.
Try it
The live demo loads a synthetic four-PDF pack in one click. Please use that pack, not real policies. The code is MIT-licensed on GitHub, and the step-by-step guide covers a local install and what to test. If you find a bug, open an issue on GitHub. If you work in claims and see a gap, leave a comment or message me on LinkedIn.
The notes
- Before a motor claim can move, someone has to dig through the policy PDFsA claims handler needs about a dozen facts from four or five PDFs before a motor claim can move. What PolicyTrace changes for that first step, what it can't do yet, and how an insurer could test it.
- Try PolicyTrace yourself: a step-by-step guideRun PolicyTrace in one click on Hugging Face, or locally in about eight minutes. Then check it the way a sceptic would, and break it on purpose.
- I masked the personal data. Then I found an unmasked copy.Masking personal data before the model sees it isn't enough if the same container keeps the unmasked text on disk. How I found it, the fix, and what masking still misses.
- I swapped the AI model and every test still passed. That was the problem.Unit tests protect the code around a model. They can't tell you whether a new model still gets the answers right, and that's the change most likely to break things.
- The right words. The wrong driver.A highlight can quote the right words from the wrong person. A text-match score can't tell the difference; the matcher has to know whose row it is.
- What happens to a PDF inside PolicyTraceThe model's job in PolicyTrace is narrow: read one masked document and fill in a form. Everything that decides whether you can trust the result happens after it.
- Two documents, one policy, four disagreementsMy first run reported four conflicts between two policy documents. Two were double spaces, and two led to a real merge bug that a quiet merge would have hidden.
AI engineer with 14 years across machine learning and applied AI: prompt design, document AI, LLM workflows and the review loops that make them safe to ship. I build real systems and write up what breaks along the way.
Get the next note by email
One short email when something ships or breaks. Free.