The invoice said £244.80. Why did the AI tool show £0?
Who chose zero? I did. Four rules for a failed AI request, replayed on 81 tasks: what each one lets through, and what it costs.
All notes
Field notes
Short, first-person notes from building AI systems in public, newest first.
Semantic caching, explained: when an AI can reuse an answer
A semantic cache reuses an AI answer when a new question is close in meaning to an old one. The real test is whether that answer is still right for this customer, today.
Version 1 let an AI agent decide. Version 2 lets written rules decide.
A better model removes the model's excuses. What's left is the design: if a check has no way to say it didn't run, it will be reported as clean.
I didn’t need the expensive model. I needed to know when to call it.
On a £100.01 refund, a cheap AI asked Finance for a ticket twice, and once just said the job was done. I tested two ways to tell when an answer needs a second opinion.
Before you quote a UK company, know what wasn’t checked
An underwriter needs a route and its reasons before quoting a company, plus an honest list of what wasn’t checked. A tool that hides its gaps is worse than none.
Try the UK Company Pre-Screen yourself
Two saved cases teach the difference between a check that couldn’t run and a finding on the record. Then screen your own company, or make it fail on purpose.
My AI judges agreed with themselves. Not with each other.
Each judge mostly agreed with itself. On borderline cases, they often didn't agree with each other, and that difference alone moved a pass rate by 27 points.
Before a motor claim can move, someone has to dig through the policy PDFs
A claims handler needs about a dozen facts from four or five PDFs before a motor claim can move. What PolicyTrace changes for that first step, what it can't do yet, and how an insurer could test it.
Try PolicyTrace yourself: a step-by-step guide
Run PolicyTrace in one click on Hugging Face, or locally in about eight minutes. Then check it the way a sceptic would, and break it on purpose.
I masked the personal data. Then I found an unmasked copy.
Masking personal data before the model sees it isn't enough if the same container keeps the unmasked text on disk. How I found it, the fix, and what masking still misses.
I swapped the AI model and every test still passed. That was the problem.
Unit tests protect the code around a model. They can't tell you whether a new model still gets the answers right, and that's the change most likely to break things.
The right words. The wrong driver.
A highlight can quote the right words from the wrong person. A text-match score can't tell the difference; the matcher has to know whose row it is.
What happens to a PDF inside PolicyTrace
The model's job in PolicyTrace is narrow: read one masked document and fill in a form. Everything that decides whether you can trust the result happens after it.
Two documents, one policy, four disagreements
My first run reported four conflicts between two policy documents. Two were double spaces, and two led to a real merge bug that a quiet merge would have hidden.
My AI agent called a fined bank “medium risk”. No model made that call.
A schema checks the shape of an answer, not whether it is true. How a default value hid two failures in my agent, and the rules I follow now.
Only one of my PDF problems was in the model
When a PDF extraction goes wrong, the model is the obvious suspect. In my system, four of the five real problems were in the ordinary code around it.
Nothing found isn't the same as nothing checked
A check that never ran reported every director as clean. The fix wasn't a better model; it was making "couldn't check" an answer of its own.