PolicyTrace Evidence and Provenance: Why JSON Is Not Enough

Problem

The answer is not finished until it can be traced.

A lot of AI extraction demos stop when the model returns JSON. That is useful for a prototype, but it is not enough for a workflow where a reviewer has to approve, reject, or override a field.

If the system says the policy starts on 2026-04-15, the reviewer needs the next question answered immediately: where did that value come from?

Failure example

Output-only JSON creates a trust gap.

{
  "start_date": "2026-04-15",
  "class_of_use": "Social",
  "voluntary_excess": "250"
}

{
  "field_path": "policy_header.period_of_cover.start_date",
  "value": "2026-04-15T00:00:00",
  "matched_text": "15/04/2026 at 00:00 hours",
  "source_filename": "Certificate.pdf",
  "page": 1,
  "bbox": [18.2, 31.4, 62.9, 34.1]
}

Architecture concept

PolicyTrace uses two representations of the same fact.

A canonical value and a source phrase are not the same thing. The canonical value is shaped for systems. The source phrase is shaped for review.

Dates are the easiest example. A PDF may say 15/04/2026 at 00:00 hours. The Golden Record may store an ISO-like value. If the system tries to find the normalized value in the PDF, it may fail. If it preserves the original phrase, provenance matching has something real to locate.

Important boundary

The model does not get to invent coordinates.

This is the key design decision. PolicyTrace does not ask the LLM to say "page 3, x 42, y 18" and hope that geometry is real. The model supplies typed values and source phrases. The system builds geometry from Docling and matches after extraction.

That separation matters because coordinates are an artifact problem, not a language problem. The PDF parser owns text and layout. The model owns candidate extraction. The provenance layer connects them.

Real walkthrough

What happens when a reviewer clicks a field.

The review UI starts with the Golden Record fields on one side and the source PDF on the other. Each field can carry provenance: the matched text, source filename, page, bounding box, and match score.

When a field is selected, the frontend finds its FieldProvenance, switches to the right PDF, scrolls to the page, and overlays the percentage-based bbox on the rendered PDF page. The reviewer is not forced to trust a JSON blob. They see the answer and its source together.

Source honesty

Provenance is review evidence, not legal proof.

It is tempting to oversell provenance. PolicyTrace should not do that. A matched highlight means the system found text that supports a field. It does not mean the business interpretation is automatically correct, or that the workflow has legal-grade audit controls.

The current implementation is honest about those boundaries. If no good match exists, the field can remain without location data. If the document type is broad boilerplate, such as a Policy Booklet, it can be excluded from matching to reduce false positives. If production required stronger guarantees, the next layer would be evaluation, audit logging, reviewer approval history, and retention policy.

Reusable pattern

The same evidence pattern travels beyond insurance.

PolicyTrace uses UK motor insurance because the document pack makes the problem visible: schedules, certificates, statements, and policy wording overlap. But the evidence pattern is broader.

Any AI system that extracts from documents should separate the value used downstream from the evidence used for review. Lending packs, claims files, compliance evidence, onboarding documents, audit bundles, and healthcare forms all need the same split.