Part of the PolicyTrace series

Documents System note

What happens to a PDF inside PolicyTrace

The model's job in PolicyTrace is narrow: read one masked document and fill in a form. Everything that decides whether you can trust the result happens after it.

Teja Sagiraju · Updated · 5 min read
The short version
• One upload, followed through PolicyTrace's code file by file: parsing, masking, classifying, extracting, merging, and tracing each value back to the page.
• The model's job is narrow: read one masked document and fill in a typed form. Everything after that is ordinary code.
• The unmasked page text is copied off before masking, so highlights land on the real words. That one branch is the part worth understanding.

When I explain PolicyTrace to someone, I usually draw a straight line: PDFs in, a checked record out, a review screen at the end. Reading the code again, the real shape is a straight line with one branch, and the branch is the part worth explaining.

PolicyTrace reads a UK car insurance policy pack, usually four PDFs, and returns one record of the facts a claims handler needs, with every value linked to its line in the PDF. This post walks one upload through the code, file by file.

One upload through PolicyTrace: four PDFs go to Docling; a copy of the unmasked page text is kept as the ProvenanceCorpus; Presidio masks names; gpt-oss-20b classifies each document and gpt-oss-120b fills a typed form; then PolicyArbiter merges, build_provenance finds each value on the page, and the review screen shows it.
One upload through the code. Amber steps call a model; the dashed box is the copy taken before masking.

Masking sits in front of the model, not in front of everything

Every PDF goes through _process_internal() in src/agents.py. Docling, an open-source PDF parser, converts it to Markdown. Then PIIMasker in src/privacy.py, a thin wrapper around Microsoft's Presidio, masks personal data: names, phone numbers, emails, most place names and a few other types become tokens like MASKED_PERSON_1. Postcodes aren't among them, because Presidio has no UK postcode rule. Only that masked text goes to the model, which runs on Groq.

The branch happens just before masking. The same Docling output is used to build a ProvenanceCorpus: every text element on every page, with its position. The comment above that line says doing it before masking is "critical for accurate fuzzy match". Highlights are drawn on the real text of the page, and the matched text a reviewer sees is the real wording, not tokens.

Two details in the masking are worth knowing:

  • Dates are not masked. mask_dates is false in config/settings.yaml, because a motor policy is mostly dates and the extraction can't work without them. That means dates of birth reach the model. The privacy post lists everything else that does.
  • Names come back as tokens. mask() also returns a map from each token back to the original name, and nothing uses it yet. So the record holds MASKED_PERSON_1, not a name, and names never get a highlight, because a token never appears on the page. Tokens numbered separately per document also caused a driver-matching bug, now fixed; that story is here.

Classify first, then one prompt per document

A policy pack is four different documents, so I don't send them to the model as one blob. classify_document() sends the first 4,000 characters to openai/gpt-oss-20b and asks for a single word: Schedule, Certificate, StatementOfFact, PolicyBooklet or Unknown. A keyword count runs alongside it ("persons entitled to drive" means Certificate, "premium payable" means Schedule). If the model's answer isn't one of the five words, the keywords decide. If both answer but disagree, the model wins and the disagreement is logged.

The document type picks the prompt from config/prompts.yaml, which is on version v2. The Schedule prompt is long and specific: fire and theft excesses are separate fields, dates of birth are dates and not ages, return null for anything that isn't there. The Certificate prompt cares about the legal-use clause and whether you can drive other cars, and leaves the money fields empty.

One thing surprised me when I traced it through src/pipeline.py. Only the first Schedule and the first Certificate go into the merge. The Statement of Fact is extracted, but the result is logged as "not used in merge" and dropped; its text still goes into the provenance corpus, so it can supply highlights. The policy booklet is capped at 20 pages and left out of matching completely, because boilerplate matches everything.

The model fills in a form

The extraction call goes through Instructor, a library that makes a model's answer fit a Pydantic model: a Python class that lists each field and its type. Here that class is UKMotorGoldenRecord in src/schema.py. Nearly every field is optional, so a Certificate can return a record that's mostly empty without the model being pushed to invent premiums. If the answer doesn't fit, Instructor retries twice (max_retries: 2).

The prompt also asks for field_citations: a map from each field to the exact phrase in the document. The record might store 2026-04-15T00:00:00 while the PDF says "15/04/2026 at 00:00 hours". The citation keeps the second version, so the matcher has something real to search for. The field is declared with Field(exclude=True), so it helps find the line without ever appearing in the final record.

The gpt-oss models think before they answer, and with the default token budget a full Schedule came back cut off; _reasoning_kwargs() now sets a 4,096-token budget and low reasoning effort. That one was the only problem I found in the model call itself, as I wrote here.

Trust is decided after the model

Everything after the model call is ordinary code:

  • PolicyArbiter in src/arbiter.py merges the Schedule and Certificate records. The Schedule wins, except for class of use and driving other cars, and every disagreement is recorded as a conflict.
  • build_provenance() in src/provenance.py walks the merged record and fuzzy-matches each value, or its citation, against the unmasked corpus. Where that went wrong has its own post.
  • POST /api/process in src/api.py writes the record, the highlights, the conflicts and an empty review file into a session folder, and the React review screen reads them from there.

That split is the reason the project exists. Which document wins, where a value sits on the page and whether a person agrees with it are all decided somewhere I can read, test and change without touching a prompt.

Limits

This is the shape of one demo system on a synthetic pack of native PDFs. Scanned documents would add OCR in front of Docling, and with it a new kind of error. The classifier has only ever seen four documents.

The decision behind it

  • Decision: the model reads one masked document at a time and fills a typed form; merging, locating and reviewing are plain code.
  • Alternative rejected: one big prompt over the whole pack that returns the final record, coordinates included. Simpler to write, impossible to test piece by piece.
  • What it cost: more moving parts to keep in step: the schema, the prompts, the merge rule and the matcher all change together.
  • When to revisit: if a model can return reliable page positions itself, the matcher could become a check rather than the source of truth.
Go further
Code: PolicyTrace on GitHub. src/pipeline.py is 131 lines and reading it next to src/agents.py covers almost everything above
Demo: the live demo runs the same code on the synthetic pack, in about three minutes
Related: Two documents, one policy, four disagreements · The right words. The wrong driver.

Where does your own pipeline draw the line between the model's job and the code's?

First published 17 Oct 2026. Updated 25 Oct 2026: new phone-readable diagram, terms explained, and the driver-matching story moved to its own post.

Written by Teja Sagiraju

AI engineer with 14 years across machine learning and applied AI: prompt design, document AI, LLM workflows and the review loops that make them safe to ship. I build real systems and write up what breaks along the way.

Connect

Get the next note by email

One short email when something ships or breaks. Free.