What I Would Improve Before Taking PolicyTrace to Production

Core thesis

A good AI demo should make the production gaps visible.

PolicyTrace already proves the core shape of a reviewable Document AI workflow: parse PDFs, protect sensitive data before model calls, extract typed records, arbitrate conflicts, map evidence, and let a reviewer inspect the result.

That is a strong foundation, but it is not the same thing as production readiness. Production means the system has to protect real documents, survive operational load, explain changes over time, and give teams a way to own the workflow after launch.

What already works

PolicyTrace proves the workflow, not the enterprise shell around it.

The project has the pieces that make the demo worth hardening. It separates parsing from extraction, extraction from arbitration, arbitration from review, and review from downstream trust. That separation gives a production team something real to build around.

The important thing is to keep the claim honest. The current app uses local session folders, session review state, synchronous processing, and public-demo safety rules. Those choices are sensible for a reference project, but they should be replaced or governed before production use.

Identity and roles

The first production question is who is allowed to do what.

A public demo can let anyone upload synthetic PDFs. A production workflow cannot. The system needs authentication and a role model before it accepts real insurance documents.

At minimum, I would separate uploaders, reviewers, administrators, and auditors. Reviewers may verify and override fields. Auditors may inspect decisions without changing them. Administrators may configure retention, model settings, and queues. That role split makes review accountable instead of anonymous.

Storage and retention

Local session folders are fine for a demo, not for real records.

The current API stores uploaded PDFs and session outputs under output/sessions, and old sessions are deleted on startup when they pass pipeline.session_ttl_days. That is useful demo hygiene. It is not a durable storage policy.

For production, I would split metadata and artifacts. Session metadata, review state, conflicts, model versions, and audit events belong in a database. PDFs, rendered pages, parsed Markdown, and other large artifacts belong in object storage with retention controls.

I would also turn debug output off by default in production. Debug artifacts are valuable while building, but Markdown, masked Markdown, extraction JSON, and metrics need the same privacy treatment as the original documents.

Background jobs

Extraction should move out of the request path.

The README is honest that public demo extraction is synchronous and can take 30 to 90 seconds. That is acceptable for a demo where one person is trying the workflow. It is not the right shape for production.

I would move document processing into background jobs. The API should accept an upload, create a job, return a job ID, and let the UI poll or subscribe for status. Workers can handle retries, page caps, model timeouts, and partial failure without locking the request.

Evaluation and accuracy

A production extractor needs tests for answers, not just code paths.

PolicyTrace already has deterministic tests around arbitration. For production, I would add an evaluation dataset that measures extraction quality across representative policy packs.

The dataset should include clean cases, conflict cases, missing fields, low-quality scans, unusual wording, date and currency formats, driver name variations, and fields that should not be extracted from boilerplate. The goal is not one accuracy number. The goal is to know which parts of the workflow are improving or regressing.

Monitoring and cost

Operational signals should be designed before launch pressure arrives.

Document AI has several failure modes that do not look like ordinary web app errors. A model may return valid JSON that is wrong. A field may be extracted without useful evidence. A prompt change may improve one document type and break another. A large PDF may push latency or cost beyond the expected range.

I would monitor the full workflow: upload volume, page counts, conversion failures, model latency, token usage, retry counts, provenance coverage, conflict frequency, review overrides, and cost per completed pack.

Audit trails

The final record should come with its decision history.

PolicyTrace already has the ingredients for a useful audit story: source PDFs, field citations, conflict entries, chosen winners, and reviewer actions. In the current demo, those are session artifacts. In production, they should become durable history.

An audit trail should record the original source values, model and prompt versions, arbiter rule version, selected winner, reviewer identity, override value, timestamp, and deletion or export actions. That history is what lets teams answer not just "what is the value?" but "how did this value become trusted?"

Production roadmap

I would harden PolicyTrace in layers, not all at once.

The right production path is staged. First, make access and storage safe. Then make processing reliable. Then make quality measurable. Then add operational feedback loops for cost, latency, and reviewer outcomes.