What happens to a PDF inside PolicyTrace

When I explain PolicyTrace to someone, I usually draw a straight line: PDFs in, JSON out, a review screen at the end. The early write-ups on this site went the other way, with a seven-layer architecture map. Re-reading the code this week, I don't think either picture is quite right. The real shape is a straight line with one branch, and the branch is the part worth explaining.

This post walks one upload through the code, file by file.

Masking sits in front of the model, not in front of everything

Every PDF goes through _process_internal() in src/agents.py. Docling converts it to Markdown. Then PIIMasker in src/privacy.py, a thin wrapper around Microsoft Presidio, swaps names, phone numbers, emails, most place names and a few other entity types for tokens like MASKED_PERSON_1. Postcodes aren't among them: Presidio has no UK postcode rule. Only that masked text goes to Groq.

The branch happens just before the masking. The same Docling output is used to build a ProvenanceCorpus: every text element on every page, with its bounding box. The comment above that line says doing it before masking is "critical for accurate fuzzy match". The idea is that highlights are drawn on the real text of the page, and the matched text a reviewer sees is the real wording, not tokens.

Two details in the masking are worth knowing about:

• Dates are not masked. mask_dates is false in config/settings.yaml, because a motor policy is mostly dates, and the extraction can't work without them. That means dates of birth reach the model.
• The token map is thrown away. mask() returns the masked text and a map back to the original values, and agents.py keeps it as _token_map and never uses it. So a name that Presidio catches comes back from the model as MASKED_PERSON_1, not as a name. Tracing it through, that has two consequences I hadn't thought about, and a fresh run confirmed both. A masked name can't be found in the unmasked corpus, so names never get a highlight. And the arbiter's fuzzy driver matching compares tokens that were numbered separately in each document. On that run the Schedule's drivers came back as MASKED_PERSON_2, _3 and _4, while the Certificate's are most likely _1, _2 and _3. Any two of these tokens score 93 out of 100 on the name match, above the 85 threshold, so a driver can be paired with the wrong person. That was a bug, and it's fixed: the masker now gives the same person the same token in every document of a pack, and masked names must match exactly. On the same demo pack, the false main-driver conflicts are gone.

Classify first, then one prompt per document

A policy pack is four different documents, so I don't send them to the model as one blob. classify_document() sends the first 4,000 characters to openai/gpt-oss-20b and asks for a single word: Schedule, Certificate, StatementOfFact, PolicyBooklet or Unknown. A keyword count runs alongside it ("persons entitled to drive" means Certificate, "premium payable" means Schedule). If the model's answer isn't one of the five words, the keywords decide. If both answer but disagree, the model wins and the disagreement is logged.

The document type picks the prompt from config/prompts.yaml, which is on version v2. The Schedule prompt is long and specific: fire and theft excesses are separate fields, dates of birth are dates and not ages, return null for anything that isn't there. The Certificate prompt cares about the legal-use clause and whether you can drive other cars, and leaves the money fields empty.

One thing surprised me when I traced it through src/pipeline.py. Only the first Schedule and the first Certificate go into the merge. The Statement of Fact is extracted, but the result is logged as "not used in merge" and dropped. Its text still goes into the provenance corpus, so it can supply highlights. The Policy Booklet is capped at 20 pages in Docling, and it's left out of matching completely, because boilerplate matches everything.

The model fills in a form

The extraction call goes through Instructor with response_model=UKMotorGoldenRecord, the Pydantic model in src/schema.py. Nearly every field is Optional, so a Certificate can return a record that's mostly empty without the model being pushed to invent premiums. If validation fails, Instructor retries twice, which is max_retries: 2 in the settings.

The prompt also asks for one more thing: field_citations, a map from each field path to the exact phrase in the document. The record might store 2026-04-15T00:00:00, while the PDF says "15/04/2026 at 00:00 hours". The citation keeps the second version so the matcher has something real to search for. The field is declared with Field(exclude=True), so it helps provenance without ever appearing in the Golden Record JSON.

Since the Groq model change I also had to add _reasoning_kwargs(). The gpt-oss models think before they answer, and with the default token budget a full Schedule came back cut off. A 4,096-token budget and reasoning_effort: "low" fixed it.

Trust is decided after the model

Everything after the model call is ordinary code:

• PolicyArbiter in src/arbiter.py merges the Schedule and Certificate records. The Schedule wins, except for class of use and driving other cars, and every disagreement is recorded as a conflict. I wrote up what that caught on a real run separately.
• build_provenance() in src/provenance.py walks the merged record and fuzzy-matches each value, or its citation, against the unmasked corpus.
• POST /api/process in src/api.py writes the record, provenance, conflicts and an empty review_state.json into a session folder, and the React UI reads them from there.

That split is the reason the project exists. The model's job is narrow: read one masked document and fill in a typed form. Which document wins, where a value sits on the page and whether a person agrees with it are all decided somewhere I can read, test and change without touching a prompt.

If you want to follow along

The pipeline is short. src/pipeline.py is 131 lines, and reading it next to src/agents.py covers almost everything above. The code is on GitHub, and the live demo runs the same code on a synthetic four-PDF pack. A run takes about three minutes on Groq's free tier.