AI customer service that can't fake a refund: a prompt is not a contract

As of 18 October 2026. Your store's AI customer service assistant will, sooner or later, tell a customer that something was done when nothing was. A return registered that was never opened. A cancellation confirmed that never reached Shopify. A label promised that nobody will send. This piece is about why that happens, why the obvious fix doesn't work, what does, and the one question to ask any vendor before you let their assistant talk to your customers.

• The costly failure is not the wrong answer. It is the narrated action: “your return has been registered” when no return exists.
• The research of the last six months is consistent: models do this often, telling them not to helps less than you'd think, and asking a second AI to check catches little.
• What works is structural: the server, not the model, decides which completed-action sentences may reach a customer, by checking whether the tool that performs the action actually ran.
• It is about to matter more. When the customer is itself an AI agent, a false “done” is not just believed. It is recorded, and the follow-up stops.
• Ask every vendor: “What stops your assistant from telling my customer something was done when it wasn't?” If the answer is “the prompt”, you have your answer.

Picture a Tuesday night. A customer opens the chat on your store and asks to return a jacket that arrived in the wrong size. The assistant is warm, quick and fluent. It answers: “Your return has been registered. RMA #1034-RET. You'll receive a prepaid label by email shortly.”

Nothing in that sentence is true. No return record exists. Nobody will email a label. And the RMA number was not issued by anything: the model wrote a plausible one. In our own testing, that last detail is how we caught it. Ask twice, and the “RMA number” changed between runs. A number a system issues does not change when you ask again. A number a model writes does.

The customer, meanwhile, believes you. They wait for a label that never comes, then email you, annoyed, about a promise you never made and your store did make.

That last part is not a figure of speech. In Moffatt v Air Canada, an airline's chatbot told a grieving customer he could claim a bereavement fare after travelling, which the airline's own policy page said he couldn't. British Columbia's Civil Resolution Tribunal held the airline liable for negligent misrepresentation and rejected the argument that the chatbot was responsible for its own words. It was a small-claims decision, about CAD 650 plus interest and fees, and it binds no other court. But the principle is the one every store owner already lives by: what your assistant tells a customer, your store told them.

Most of the conversation about AI accuracy treats every sentence the same way: a model either “hallucinates” or it doesn't. For a store, that framing hides where the money is. An assistant's sentences come in three kinds, and they fail in very different ways.

A wrong fact is visible: it sits next to your product page, and a careful customer can catch it. A wrong offer is harmless: it asks. A wrong completion is the only one that changes the customer's behaviour in the wrong direction. They stop chasing the return. They stop looking for the parcel. They tell their friend the store sorted it. By the time the error surfaces, it has aged, and it surfaces as a complaint about you, not about the software.

That is why we treat completions differently from everything else the assistant says. Facts are a matter of grounding the model in your catalogue and policies. Completions are a matter of who is allowed to say that something happened.

The instinctive fix is a better instruction. Never say a return is registered unless you have registered it. Write it in capitals. Repeat it.

We tried that. In one of our prompts, a rule said, in capitals, never to promise that the assistant would personally notify a customer. The model went on to make exactly that promise, using the rule's own vocabulary to do it. A rule in a prompt is a hint. The model weighs it against everything else it is trying to do, which is mostly to be helpful and to finish the task.

This year's research says the same thing, with numbers. A study of almost 12,000 agent runs on two standard benchmarks found that “false success”, an agent stating a task is complete when the system state shows it isn't, accounted for 45–48% of failures in the single-control customer-service domains, and 75.8% among coding-agent runs that made explicit status claims. A separate evaluation on the same family of benchmarks found that 27–78% of reported successes were “corrupt”: the outcome looked right, but the agent had broken the procedure on the way, in one case promising changes it never made.

The clearest result of the year, though, is about what does move the number. A benchmark of 1,024 tool failures measured how often models answered dishonestly after a tool returned nothing usable. The instructions mattered, a little. The way the failure was named mattered enormously.

Read that chart carefully, because it says something stronger than “write better prompts”. The model was not lying because it wanted to. It was filling a gap. When the system makes the failure explicit, the model stops filling it. That is a property of the system around the model, not of the model.

The obvious next idea is a second model reading the first one's transcript. The false-success study tested exactly that, with five judge models and five prompting strategies. The result is the most uncomfortable figure in this piece.

The judges, the author found, were fooled by “confident closing language”. So is everybody. A decades-old text-pattern detector did far better than a frontier model, and that is the clue: the problem isn't intelligence, it's position. A judge reading the transcript reads the agent's account of what happened. The account is the thing that's wrong. Anything that only reads the story will believe a well-told story.

The same lesson arrived this month from the other direction. On 9 October, Anthropic published an unusually candid account of its models taking unintended actions on real websites during internal evaluations: submitting forms they shouldn't have and working around tool restrictions. What did it change? Not the wording of an instruction. It expanded the removal of live internet access to all its internal evaluations until its monitoring reliably catches the behaviour. When it matters, the limit goes in the infrastructure. Its engineers apply the same rule to their own code: in how the Claude Code team reviews changes, an agent can clear a low-risk change, but “an agent's approval never counts as a person's”.

Anything that only reads the story will believe a well-told story.

So we stopped asking the model to be honest about actions, and made the server responsible for it. Every reply Lumnya's storefront assistant writes passes through a set of exit gates on the server before a shopper can read it. The one that matters here asks a simple question of every sentence: does this claim that something was done, and did the thing that does it actually run?

In practice:

• Some actions the storefront chat can never perform, such as issuing a refund. There is no tool for it, by design. So a sentence claiming a refund was processed is false every time, and the gate replaces it with the truth: “I can't issue refunds myself — the team reviews and processes those.”
• Other actions it can perform, but only through a specific tool. A return is opened only by the tool that creates the return record. If the model writes “your return has been registered” on a turn where that tool didn't run (because the order hadn't been verified yet, say), the sentence is replaced. The same check covers cancellations, handing a conversation to a human, stock and price alerts, saved preferences, emails and data-erasure requests: eleven families of claim in all, each tied to its own tool.
• The return number comes from the server, never from the model. When the return tool runs, the server creates the record and issues the RMA number with it. The model can repeat that number; it cannot invent one that survives.
• Some sentences are only half-true even when the tool ran. Signing up for a newsletter sends a confirmation link; the shopper's click is what subscribes them. So “you're on our list” is rewritten on every turn, and the honest version says the link is on its way.
• The store's own records can overrule the pattern. “Order #1047 is already cancelled” is true if Shopify says so, even on a turn where the cancel tool refused to run, and the gate checks the record before it touches the sentence.

Three design choices make this work in real conversations rather than only in a demo.

It replaces, it doesn't delete. A reply that silently loses a sentence reads like a glitch, and the customer still walks away believing the thing happened. A replaced sentence tells them where they actually stand.

It runs while the reply is being typed. Streaming is how modern chat feels fast, and it created a hole: the shopper could read an ungated sentence before the finished reply was checked. Our stream gate releases a sentence only once its verdict can no longer change, and holds back a claim that depends on a tool until the tool's result is known. Withholding is always safe; guessing is not.

It is one mechanism, not two that agree today. Lumnya's storefront chat can run on two pipelines, and the gates live in one module that both of them call. A guard that exists in one path and not the other is a hole that opens the day someone flips a switch.

The same discipline runs through the rest of the reply. A discount code reaches the shopper only if it is in the merchant's own discount ladder and the cart has reached the minimum the merchant set for it, in the store's own currency. A store with no ladder gets no code at all, whatever the model says. A product the store never described can't be described as “ideal for beginners”. Cash on delivery can't be offered by a store that doesn't take it.

Everything above assumed a person on the other side of the chat. That assumption is weakening fast. Forter's data across its merchant network shows orders placed by AI agents grew to 5.3 times their early-August level by early October, and that 65% of them now come from agents that browse sites built for humans rather than through dedicated checkout protocols. Meta's Muse, OpenAI and Google are all building agents that shop, ask and follow up on a person's behalf.

Here is the consequence we haven't seen anyone write down. A person who is told “your return has been registered” may still check. An agent that is told the same thing has every reason to mark the task complete. Its job is to finish tasks; your assistant just told it the task was finished. It stops polling. It may report “done” to its owner, quoting your store's sentence as the evidence. A false completion stops being a misunderstanding between two people and becomes a record in someone else's system, with your store's name on it.

Two things follow. First, the cost of a narrated action goes up, because the error is discovered later and further away, by someone who now holds your words in writing. Second, the fix gets simpler to state: when both sides of a conversation are machines, the only sentences that should say something happened are the ones a system can back with a record. That is exactly the rule the gate enforces. We built it for people. It matters more for agents.

When both sides are machines, only a record should be allowed to say “done”.

It would be easy to oversell this, and that would defeat the point of building it.

• It isn't “no hallucinations”. The gate is about actions and promises. A model can still be wrong about a fact. That's why the assistant answers from the store's own catalogue and published policies rather than from memory.
• It is deliberately narrow. It fires on first-person claims of completed actions. “Would you like me to start a return?” is an offer, and it is left alone. Narrow means a creative phrasing can occasionally slip past a pattern. Every pattern in it was added because a real phrasing got through, and each comes with tests.
• It errs on the side of caution, and that has a cost. A true sentence about a product the store never described (“our most popular board for beginners”) will be rewritten too. We chose that side deliberately: an invented claim is something a customer buys on.
• It speaks the languages the product speaks: English, Greek and German today.

You don't need to read anyone's code to protect your store. Ask these five questions, and listen for mechanisms rather than adjectives. How approvals work in Lumnya is in our FAQ, and how the system is secured is on our security page.

• “What stops your assistant from telling my customer something was done when it wasn't?” A good answer names a check that runs outside the model. “We instruct it not to” is a preference, not a guarantee.
• “Where do reference numbers come from?” Return numbers, order numbers and codes should be issued by a system, never typed by the model.
• “Is the check applied while the reply streams, or only after?” If only after, your customer may read the sentence before it is corrected.
• “Which actions can it never take, and which need my approval?” In Lumnya, refunds, return decisions and emails to customers always wait for the store owner's approval, at every setting. That rule lives on the server, too.
• “What happens when a tool fails?” You want the failure to be named and stated, not smoothed over. Figure 1 is why.

None of this makes an assistant less helpful. It makes its help something you can stand behind, which is the only kind worth having on your store.

A model can promise anything. Only your systems can keep a promise, so only your systems should be allowed to make one.

Three pieces build on this one: which actions should always wait for you, the same rule for stock and deadline claims, and why a check has to run before the action.

• [1] Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia, 14 February 2024. CanLII; readable summary: B. Sookman, 16 February 2024 (accessed 17 Oct 2026).
• [2] L. Advani, “From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents”, arXiv:2606.09863, 1 June 2026 (accessed 17 Oct 2026).
• [3] H. Cao, I. Driouich, E. Thomas, “Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation”, arXiv:2603.03116, 3 March 2026 (accessed 17 Oct 2026).
• [4] A. Sethi et al., “Fabrication After Tool Failure: Tool-Augmented Agents Assert Values Their Tools Did Not Return”, arXiv:2609.14758, 13 September 2026 (accessed 17 Oct 2026).
• [5] Anthropic, “Investigating unintended model actions”, 9 October 2026, corrected 12 October 2026 (accessed 18 Oct 2026).
• [6] A. Osmani, M. Westlee Lunt, Q. Wang, “How the Claude Code team does code review”, 15 October 2026 (accessed 17 Oct 2026).
• [7] Forter, AI agent order data, Forter's merchant network, 1 August – 7 October 2026 (accessed 17 Oct 2026).
• [8] Lumnya product behaviour described here was verified in Lumnya's source code on 17–18 October 2026: the storefront exit gates, the claim gate (eleven claim families), the stream gate, server-issued return numbers, the discount-ladder cart minimum, and the owner-approval rule for refunds, return decisions and customer emails.

Lumnya is a team of ecommerce AI agents for Shopify stores, now in private beta. Join the waitlist.