Version 1 let an AI agent decide. Version 2 lets written rules decide.

Version 1: an agent that decided

The tool answers one question for a commercial insurance underwriter before they quote: is there anything on the public record about this UK company that should stop me, or make me ask the broker first?

My first version, which I deployed on Hugging Face on a free model tier, was a research agent. It worked like an analyst would: find the company, plan which sources to read, collect them, read the news, critique its own findings, look for gaps and loop back, then write a brief. The last step, a model, also wrote the risk level. Nothing written down said what made a case "medium".

What it got wrong on live data

On 18 Oct 2026 I ran version 1 on four real companies and wrote down every mistake.

• Wilko Limited: "high risk", from a collapse that belonged to a different company which had used the same name. That's the next post in this series.
• Thames Water and Apex Building Services: it never found a company number, carried on anyway, and still reported a clean ownership register.
• Monzo: "medium risk", because a schema default filled a gap. That's My AI agent called a fined bank “medium risk”.

Was it just the model?

That's the fair objection: maybe I was blaming the design for a weak, free model. So on 19 Oct I ran version 1 on a paid model (gpt-6-luna) against version 2, on nine real companies. Before any run I wrote down each company's key fact and expected outcome. Then a separate AI agent, which hadn't built anything, rescored every output without seeing my scores.

The better model fixed the model's problems. Every company resolved, no call failed, and the writing was careful: it hedged and flagged its own contradictions. It didn't fix the design.

In all nine runs, its sanctions screen checked zero names, and every brief reported no sanctions hits. Eight also said, further down, that nothing had been screened; Linklaters' didn't. Somewhere in the plumbing a list came back empty, and the agent had no way to record "this check didn't run", so an empty result came out reading as clean. The careful writing made it worse: it read like diligence.

It wasn't alone. The FCA check said "unknown" for every firm, including an authorised bank, and the phoenix score counted the company being checked as one of its own director's failures.

On the key-fact rubric, version 1 scored 2 of 9: four facts half-found, none in full. Version 2 scored 9 of 9. Version 1 made 4 wrong statements to version 2's 2, cost about thirteen times as much per company (14.9 model calls to one), and took 89 seconds to 9. Its 8 of 9 right outcomes look close, but on Wilko and Linklaters it got there for the wrong reasons.

Version 2: rules decide, models read

Version 2, the UK Company Pre-Screen, pins the case to one company number before anything runs. It then runs 13 checks (14 when the proposal says what the company does), and each one ends in a stated result, including "couldn't check". Written, versioned rules pick one of four routes: Note only, Refer, Outside appetite or Compliance hold. One model call reads the news, to answer one narrow question: is this article about this company? Later I added agents on top that explain a case, but they can't change the route.

Then and now

Version 1, the research agent

• Good: quick to build, flexible, and its briefs read like an analyst's.
• Good: it could say something about questions nobody had written a rule for.
• Bad: no way to say "this didn't run", so gaps came out as "clear".
• Bad: a model chose the risk level, and nobody could check why.
• Bad: about 15 model calls per company, and its mistakes hid inside fluent prose.

Version 2, the Pre-Screen

• Good: every route traces back to a written rule and a source.
• Good: gaps are counted and shown on the page.
• Good: one model call per company, and the same answer every run.
• Bad: plainer output, and it only knows what the rules cover.
• Bad: every threshold has to be written down and defended, and mine are still drafts.
• Bad: the rules need maintaining as sources and news change, which the rest of this series is about.

Where I was wrong

My own scoring gave version 2 one wrong statement. The rescorer found two, both on Revolut. One is the subject of a later post; the other, a trade check that said SIC code 62090 covers e-money, I'd missed because it read plausibly. That's the case for having someone else mark your work, even when the someone is another agent.

Limits

• Nine companies, one run each: a development set, not a benchmark.
• The two sides didn't get the same input. Version 2 reads the proposal (trade, turnover); version 1 doesn't. They also ran different models.
• Version 2's rules were changed that same day, partly because of runs on these companies. Before that change its outcome score was 8 of 9.
• The rescorer is an AI agent, separate from the one that built the system, not an underwriter.

If your design has no place for "I don't know", a better model will fill that gap more fluently. Try the better model first, because it's cheap and it removes every excuse that belongs to the model. What's left is the design.

First published 7 Nov 2026.