In Compliance, a Confident Guess Is the Most Dangerous Answer

I have spent most of my career building software where being wrong is expensive: credit reporting at ClearScore, payments and logistics at Deliveroo. Environments like that teach one habit above all others. You stop trusting any output that cannot explain itself.

That habit, not any particular model, is what shaped Karavel. In compliance, the failure that costs you is not the system that admits it is unsure. It is the system that is sure, sounds right, and is wrong.

Confidently incorrect

In most software, a confidently wrong answer is a support ticket. In compliance it is the exact thing the regime exists to prevent. A tool that says “this ad is fine”, cannot say why, and turns out to be wrong has not made a small mistake. It has manufactured the precise exposure that regulators fine firms for.

The two ways of being wrong are not equal, either. Flag an ad that was fine and you cost a reviewer a few minutes. Pass an ad that was not and you can cost the firm a breach. An honest tool treats those as different problems, and worries far more about the second.

Sounding right is not being right

Here is the uncomfortable part, stated plainly for the people who have to rely on this. A language model is trained to produce fluent, plausible text. It is not trained to be correct, and it has no built-in sense of when it does not know. It will write a clean, persuasive paragraph explaining why an ad complies whether or not that is true, and the better the model gets, the more convincing the wrong answer becomes. The industry’s word for this is hallucination. Measured rates run from low single figures to a quarter of answers or worse on some finance questions.

The evidence is not subtle. In 2023 a New York lawyer filed a court brief built on cases that ChatGPT had invented. They looked real. The lawyers were sanctioned. Closer to our world, Stanford tested the leading legal-research tools, the ones sold as “hallucination-free” because they look answers up in real databases, and found they still got the law or the citation wrong between one in six and one in three of the time. The worst errors were the quiet ones: a confident citation to a real source that did not support the claim. Correctly formatted, authoritative, wrong.

So we settled one thing early and have never moved off it. The model is never the source of truth, and it never gets the last word.

The regulators are naming it

This is no longer a niche engineering worry.

FINRA’s 2026 oversight report lists AI hallucination as a compliance risk in its own right, and tells firms to wrap controls and human oversight around it. The EU AI Act puts much of financial-services AI in its high-risk tier, with hard obligations attached: meaningful human oversight rather than a rubber stamp, demonstrable accuracy, and logs detailed enough to reconstruct what the system did. The FCA has been clear it is not writing a separate AI rulebook, because the rules it already has, Consumer Duty and senior-manager accountability, apply to AI like anything else; Parliament has asked it to spell out, by the end of 2026, how a named manager stays accountable for harm an AI causes. Enforcement is going to machine scale too: the ASA now scans tens of millions of ads a year, as its 2025 report sets out.

One bar is forming across all of it. Not “usually right”, but “right, and able to show its working”. You cannot hand your accountability to a model, so a model that cannot show its working is of little use to you.

The system around the model

The model is one component. The system around it is the product, and its whole job is to keep the model honest. Roughly in order:

• Read it in context. At Karavel we judge most assets on their own; when one spans several parts, a carousel or a multi-step flow, we review them together as a single promotion, so a claim in one place can be weighed against a disclaimer in another. And when the verdict turns on something the asset cannot show, who it will be shown to, or the terms behind a headline offer, the system says what it needs rather than bluffing a false yes or no.
• Break it into claims. We pull the piece apart into the specific things it states or implies, and check each on its own. “Is this compliant” is not one question, it is dozens, and a piece judged as a single blob lets a weak claim hide behind the strong ones. Asked one at a time, nothing hides.
• Ground every finding in the rulebook. Grounding means the model cannot answer from memory; it has to look up the market rules for that product and channel and quote them back. No rule to cite, no finding, and a confident guess has nothing to cite.
• Check your own rules too, not just the regulator’s. Brand guidelines, the terms you must and must not use, your bank of pre-approved claims and any in-house policy are checked in the same pass. A promotion can be perfectly legal and still be off-brand or break an internal rule, and you want both caught at once.
• Keep a human at sign-off. The system’s verdict is a recommendation, not the decision. It lays out the findings and the evidence behind each, then a named reviewer weighs them and either signs the piece off or sends it back for changes. The decision on record is the person’s, never the model’s.

Put together, these steps share one purpose: to leave a confident guess nowhere to hide. By the time a verdict appears it has been built from individual claims, each tied to a rule the system can point to, with a person making the final call. The model never gets to wave something through on fluency alone.

The audit trail is the actual product

A compliance team does not need a verdict. It needs a defensible verdict. When a regulator, or your own board, asks “on what basis did you approve this”, the answer “the AI said so” is not weak so much as the wrong shape. The answer that holds is “here is the claim, here is the rule it was checked against, here is who signed it off and when, and here is every revision the ad moved through to get there”.

So we treat that trail as a first-class part of the product, not logging bolted on at the end. The reasoning is the deliverable. The yes or no is almost a by-product. It is also, not by coincidence, exactly what the new logging and accountability rules expect you to be able to produce.

What we do not automate

The system does the heavy, repetitive reading, surfaces issues in seconds, and applies the rulebook the same way every time, which a tired human at the end of a long day cannot honestly promise. What it does not do is take the final word. Sign-off stays with a person. That is not caution for its own sake; it is where the defensible standard sits, which is human judgement against authoritative sources. It is also the opposite of replacing the compliance team. The point is leverage: spend a good reviewer’s expertise on the calls that need it, not on the first pass over a thousand ads.

The boring answer is the correct one

The flashy pitch for AI compliance is “upload your ad, get an instant yes or no”. The correct product is slower to build and far more useful: fast where speed is safe, rigorous where rigour is required, and able, every time, to answer the only question that matters in our world, which is “says who”. A confident guess can never answer that. That is exactly why, in compliance, it is the most dangerous answer there is.

I will take the boring, correct version every time.

---

Karavel reviews financial promotions with grounded, cited, auditable AI and a human at sign-off, so every decision can show its working. If you want to see what that looks like on your own ads, get in touch.