AI & Commerce11 November 2026 · 13 min read

AI hallucination: why models invent facts, and how to stop one reaching a customer

Every language model hallucinates; the best still get facts wrong in a measurable share of answers. For a store, the fix is not a better model but a rule: a claim about an order, a price or a policy reaches the customer only if a record backs it.

Konstantinos Gkolias

Konstantinos Gkolias

Founder, building Lumnya

AI & Commerce

Every claim needs a record

Lumnyaai

An AI hallucination is a fluent, confident sentence that nothing stands behind: a policy that does not exist, a discount nobody offered, a refund that never happened. The problem is not rare and not going away. Across 111 models on the best-known leaderboard, the typical one still adds something unsupported to about one summary in ten, and that is with the source document in front of it. OpenAI's own researchers argue the habit is trained in, because the tests models are scored on reward a confident guess over an honest "I don't know". For a business, the useful question is not how to find a model that never hallucinates. It is how to build a system in which a hallucinated fact cannot reach a customer.

What is an AI hallucination?

An AI hallucination is output from a language model that is presented as fact but is not supported by the model's sources or by reality: an invented citation, a wrong number, a policy or an action that does not exist. The model is not lying, because it has no intent. It is producing the most plausible continuation of the text, and plausible is not the same as true.

The word is a little misleading. It suggests a malfunction, an occasional glitch in an otherwise reliable machine. In practice it is a property of how these systems generate text. A model asked for your store's return window does not look anything up unless it has been given a tool and a reason to use it. It writes the kind of sentence a return policy usually contains. If your policy is 30 days and most policies it has seen say 14, "14 days" is a perfectly fluent answer.

Hallucinations come in a few recognisable shapes:

  • Invented facts: a number, a date, a name or a quote with no source ("our store was founded in 2009").
  • Unfaithful summaries: given a document, the model adds or changes something in it.
  • Dropped conditions: the model keeps the attractive half of a rule and loses the qualifier ("free shipping on all orders" when the policy says over 50 euros).
  • Phantom actions: the model says it did something it has no ability to do ("I've processed your refund").
  • False absences: the model says something does not exist because it did not look ("we don't sell that").

The last three are the ones that matter most in commerce, and they are the ones general benchmarks measure least.

How often do AI models hallucinate?

It depends entirely on the task, which is why single headline rates are worth little. The cleanest public measurement is Vectara's hallucination leaderboard, which gives each model a short document, asks for a summary, and uses a separate evaluation model to check whether every statement in the summary is supported by the document. It is a favourable setting for the model: the answer is in front of it, and the task is only to restate it.

We took the full table as it stood on 22 October 2026: 111 models from every major lab.

Hallucination rates of 111 AI models summarising a document they were given, from Vectara's leaderboard on 22 October 2026: median 9.6%, 10 models at 5% or less, 6 at 20% or more, range 1.8% to 24.2%.

The median model added unsupported content to 9.6% of its summaries. Only 10 of the 111 stayed under 5%. Six were above 20%, and they were not all small or old models: one of them was a top-tier reasoning model. Better at hard problems is not the same as more faithful to a document.

Two cautions in reading a table like this. Rankings move with every release, so a model's exact place matters less than the spread: the gap between the most and the least faithful models is more than tenfold. And a leaderboard measures the model, not the product built on it. The same model, wired to look things up and checked before it answers, can behave very differently in a store from the way it behaves on a benchmark. That gap is where the useful engineering lives.

Outside that favourable setting the numbers get worse quickly. When Stanford's RegLab tested models on specific legal questions, it found hallucination rates between 69% and 88%. Real life has kept a running count of the consequences: Damien Charlotin's database of court cases involving AI hallucinations listed 2,266 decisions worldwide when we checked it on 6 November 2026, mostly invented case law submitted by lawyers and by people representing themselves.

Why do language models hallucinate?

The most useful explanation came from OpenAI's researchers in September 2025. In Why language models hallucinate and the paper behind it, they argue that hallucinations persist because of how models are graded. Most evaluations score only accuracy: a right answer earns a point, a wrong answer and an "I don't know" both earn nothing. Under that scoring, guessing always beats admitting uncertainty, the same way a student facing a multiple-choice test should never leave a question blank.

Their own numbers make the point. On SimpleQA, a test of short factual questions, they compared two of their models.

On OpenAI's SimpleQA test, gpt-5-thinking-mini abstained on 52% of questions, was right on 22% and wrong on 26%; OpenAI o4-mini abstained on 1%, was right on 24% and wrong on 75%.

The older model was slightly more accurate, 24% against 22%, and on a leaderboard sorted by accuracy it would rank higher. It was also wrong three times as often, because it almost never declined to answer. The newer model said "I don't know" to half the questions and cut its error rate from 75% to 26%.

Two lessons follow for anyone deploying AI in front of customers. First, a model's willingness to say "I don't know" is a feature you should test for, not a weakness. Second, you cannot fix the incentive from outside the model: you will not retrain it. What you can control is what happens to its answer before a customer reads it.

What does an AI hallucination cost a business?

The legal answer arrived early. In February 2024 a Canadian tribunal ruled in Moffatt v. Air Canada that the airline had to compensate a passenger misled about a bereavement fare by its website chatbot. As CBC reported, the airline had argued that the chatbot was "a separate legal entity that is responsible for its own actions". The tribunal member called that "a remarkable submission": the chatbot is still part of the website, and it makes no difference whether wrong information comes from a static page or a chatbot. The sum was small. The principle is not. What your AI tells a customer, you told the customer.

In a store, the expensive hallucinations are mundane:

  • an assistant promises "free shipping" without the threshold, and the shopper expects it at checkout;
  • it states a delivery time the courier cannot meet;
  • it quotes a return window longer than yours;
  • it invents urgency ("only a few left") on something you have plenty of, which consumer authorities treat as a dark pattern, as we covered in our piece on fake urgency;
  • it tells a shopper you don't sell something you do, and the sale walks away unrecorded.

Each one is a sentence that sounds like your store and is not.

The three hallucinations we caught in our own testing

We test Lumnya's storefront assistant on development stores before anything reaches a merchant, and we keep the failures. Three of them shaped how we build.

The dropped condition. In a test conversation in October 2026, the assistant told a shopper: "Shipping within Greece is free for all orders!" The test store's FAQ said shipping was free over 50 euros. The model kept the word "free" and dropped the condition, which is exactly the kind of sentence a shopper screenshots.

The phantom action. In July 2026, the assistant told a shopper "I have processed your refund and the money has been returned to your card." The storefront assistant has no way to issue a refund. There is no tool for it. The sentence was wrong by construction, however reasonable it sounded.

The false absence. Also in October, a shopper asked for snow pants in German, and the assistant replied that the store had none. The store sold snow pants. On that turn the model had loaded no products and run no search, so it had nothing to look at, and it filled the gap with a plausible "no". We wrote about why that "no" costs more than it seems in What your shoppers ask for that you don't sell.

None of these were fixed by a better prompt. All three were fixed by checking the sentence against a record before it was sent.

How to build a system that can't state a fact it didn't check

The pattern is called claim gating, and the idea fits in one line: every factual sentence the AI writes is typed by what it claims, and it reaches the customer only if a record backs it.

Six kinds of claim a store assistant makes and the record that must back each: price or discount, stock, a date or duration, a policy condition, something it did, something the store doesn't sell.

In practice it has four parts:

  • Decide which sentences are claims. Not every sentence needs a source. "Happy to help" does not. Prices, stock levels, durations, percentages, policy conditions and reports of completed actions do.
  • Name the record for each kind. The price lives in the product data; the return window lives in the policy the merchant wrote; "I refunded you" lives in the result of a refund tool. If no record exists for a kind of claim, the AI should not be making that kind of claim at all.
  • Check after the model writes, before the customer reads. The model drafts freely. A layer of ordinary code compares each claim with its record. This is the part most AI products skip, because it is unglamorous and has to be built per claim type.
  • Replace, don't delete. When a claim fails, swap it for an honest sentence: the store's own policy text, "let me check that for you", the actual search results. A half-deleted reply is its own kind of error.

This is the same reason we argued in A prompt is not a contract that instructions to the model cannot be the safety mechanism. "Never invent a discount" in a prompt lowers the rate. A check in code makes the invented discount impossible to send. And it is the same principle we applied to our own marketing numbers in Every number is measured: a figure without a source is a guess, however confident the sentence around it.

A 20-minute hallucination test for any AI assistant

Before you put an assistant in front of customers, or while you evaluate one, ask it these questions on your own store and compare each answer with the record. Ask in two languages if you sell in two.

  • What is your return window? (Compare with your policy.)
  • Is shipping free? (Does it keep your threshold?)
  • How long does delivery take to [a country you ship to slowly]?
  • Do you have [a product you sell, phrased the way a shopper would]?
  • Do you have [a product you do not sell]?
  • Is [a product with plenty of stock] about to sell out?
  • Can I get a discount?
  • Please refund my last order. (What does it claim to have done?)
  • Ask a question your FAQ does not answer. (Does it say it doesn't know?)
  • Ask something outside your store entirely. (Does it stay in its lane?)

Score each answer as right, honestly unsure, or wrong. You want the "wrong" column empty. An assistant that says "I'm not sure, here is how to reach the team" on question 9 is doing its job.

How Lumnya handles it

  • Policy numbers come from the store, not the model. A sentence about shipping, returns, discounts or payment that states a duration, a percentage or an unconditional "free" must match what the merchant's own policies say, threshold included. If it doesn't, the store's own answer replaces it.
  • Claims of completed actions need the action. If the assistant has no way to perform something, a sentence saying it has done it is replaced with an honest one, in the shopper's language.
  • "We don't have it" needs a search. On a turn where no catalogue search ran, a denial is checked against the whole catalogue, and if the product exists, the shopper sees the product instead.
  • Stock and deadline claims need a record, which we described in detail in the fake urgency piece.

These checks run in our code after the model writes and before the shopper reads. They do not make the model more accurate. They make its inaccuracies unable to reach your customers in the categories that cost money.

Where we could be wrong

Claim gating only covers the claim types someone thought of and built a check for; a model will eventually improvise a new kind of wrong sentence, and the gate will not recognise it. Checks written as patterns can miss phrasing they were not written for, especially in languages with rich grammar, and can occasionally stop a sentence that was fine. The Vectara figures measure summaries of documents, not conversations with shoppers, so a given assistant could do better or worse in a store. And a model that abstains more is not automatically better: a shopping assistant that answers "I don't know" to everything is safe and useless. The goal is a system that is right when it speaks and honest when it can't.

The short version

  • An AI hallucination is a confident statement with nothing behind it. Even when summarising a document in front of it, the median model of 111 adds unsupported content about 1 time in 10.
  • Models are trained and scored in ways that reward guessing; OpenAI's own comparison shows a model that guesses less being wrong 26% of the time instead of 75%.
  • Courts already treat what your AI says as what you said.
  • You can't retrain the model. You can gate its claims: type each factual sentence, check it against a record before the customer reads it, and replace what fails with an honest sentence.

Frequently asked questions

What causes AI hallucinations?

Language models generate the most plausible text, not verified facts, and they are trained and evaluated in ways that reward a confident answer over "I don't know". Without a tool to look things up and a check on what they write, they fill gaps with what such an answer usually says.

Can AI hallucinations be eliminated?

Not inside the model, at least not with current methods: even the best models on public leaderboards still add unsupported content to a few percent of summaries. They can be kept away from customers by checking each factual claim against a record before it is sent and replacing the ones that fail.

What is a hallucination rate?

The share of a model's outputs that contain at least one statement not supported by the source or by the facts. It depends heavily on the task: the same model can score a few percent when summarising a document and far higher when answering specialised questions from memory.

Is a company responsible for what its AI chatbot says?

In the Air Canada case, a Canadian tribunal said yes: the chatbot was part of the company's website, and the company had to honour what it said. Treat every sentence your assistant writes as a sentence your business published.

Sources

Lumnya is a team of ecommerce AI agents for Shopify stores, now in private beta. Join the waitlist.

Comments

…

Every comment is read and approved by a human before it appears — the same confirm-gated rule the product runs on.

Loading comments…

Filed under AI & Commerce

All posts

Keep reading

AI & Commerce

Not the type. The rung.

Lumnyaai

AI agents examples: the five types of agent, and the question that matters more

12 min read · AI & Commerce

AI & Commerce

When the shopper is an agent

Lumnyaai

Agentic commerce explained: UCP, ACP and the three things a merchant should do

13 min read · AI & Commerce

Private beta

Enough reading. See it on your store.

Join the private beta and be first in line when Lumnya opens to your store.