Most chatbot ROI numbers are assumptions. We deleted ours.

As of 23 October 2026. Until eleven days ago, the home page of Lumnya's own dashboard, in the private beta we are building, showed numbers we had never measured. “Hours saved.” “Lumnya revenue.” A “store health” score. A sales forecast. Small trend lines that always rose. They looked good, and they were the kind of chatbot ROI numbers every software product ships to feel alive on day one. They were also, every one of them, an assumption with a chart attached. On 11 October we deleted them all. This is what they were, why numbers like them are so hard to question, what replaced them, and four questions you can ask of any figure on any dashboard, ours included.

• Most “AI impact” figures are a real count multiplied by an assumed constant: chats × 5 minutes, sales × 30%, orders × 1.2. The constant decides the result.
• These numbers are convincing for a specific reason: because the count underneath is real, the estimate rises and falls exactly like the truth. We call it borrowed credibility.
• People are poor judges of AI's effect even on their own work. In one controlled study, experienced developers believed AI had made them 20% faster; measured, they were 19% slower.
• Deleting the estimates was the easy half. The harder half was finding real counts that were counted wrong: a read limit shown as a total, a week keyed to the wrong time zone, a shopper's question counted as a discount given.
• Before the busiest weeks of the year, ask four questions of every number you plan to act on. The grid is below.

On 11 October the rule became: nothing shown to a merchant may be made up. Then we went through every page of the product, figure by figure, to see what that rule would cost. It cost most of the home page.

The home page was only the start. The same sweep, and a second one the next day walking every page of the live app, found the same pattern almost everywhere a number appeared.

None of this was meant to deceive. Each figure was a placeholder that made an empty dashboard look alive, written when there was no data yet, and then never replaced. That is how it happens in most products. But look at the right-hand column again. Every one of those numbers has a fixed value inside it, and that value, not the store, decided what the page showed. A store with a terrible week would still have “saved” five minutes per chat.

If those numbers were obviously fake, nobody would have trusted them, us included. The reason they survive in so many AI products is subtler, and as far as we can find, it has no name. So we gave it one.

Take “hours saved”. It is the number of chats multiplied by five minutes. The number of chats is real. So when a store has a busy week, “hours saved” goes up. When the store is quiet, it goes down. It moves exactly in step with reality, and that movement is what makes it feel measured. The estimate borrows the credibility of the count underneath it. Every glance at the dashboard confirms it: busy week, more hours saved. Of course.

But the movement is the only part that's real. The level, the number of hours you'd quote to your accountant or use to decide whether to hire, is set entirely by the constant. Change five minutes to two and the store has saved 60% fewer hours, without a single conversation changing. Change it to eight and it has saved 60% more.

Now ask who chooses the constant. It is the vendor, the one party whose renewal depends on the number being large. Nobody needs to act in bad faith for this to go wrong. The constant is picked once, on a day with no data, by someone who believes in the product, and it's natural to pick it generously. After that it never has to be defended again, because the line it draws keeps moving in such a convincing way.

An estimate built on a real count moves like the truth. Only its size is invented, and the size is the part you act on.

There is an old name for the general problem. Goodhart's law, in the form most people quote, says that “when a measure becomes a target, it ceases to be a good measure.” AI products have a sharper version of it. When a product is judged on the impact it reports, and the product also chooses the constant inside the report, the reported impact drifts up and away from the real one. The only defence we know is structural: the product doesn't get to report anything it didn't count.

You might think a store owner would feel the difference. If the assistant really saved twenty hours a month, wouldn't you notice? The best evidence we know of says: not reliably, not even when the work is your own.

In a randomised controlled trial published by METR in July 2025, 16 experienced open-source developers worked on 246 real issues in their own projects, each issue randomly allowed or disallowed AI tools. Before starting, the developers expected AI to speed them up by 24%. After finishing, they still believed it had sped them up by 20%. Measured, they took 19% longer with AI than without it.

The study has real limits. It is small, it used early-2025 tools, it measured software work rather than customer service, and METR itself has since published newer results and describes these as out of date. We don't cite it to say AI slows anyone down. We cite it for the gap between the two bars on the right. Skilled people, timing their own work, could not feel the direction of the effect. A store owner glancing at a dashboard tile is in a much weaker position, and the tile's constant fills exactly the space where their judgement can't reach.

Removing the invented numbers took a day. The harder discovery came the day after, walking every page of the live product: some of the numbers that were counted had been counted wrong. Not invented, not assumed. Honest in intent, and still false.

These are worth dwelling on, because they are the errors a careful merchant can't catch from the outside. A wrong time zone looks exactly like a right one. A capped total looks exactly like a total. The defence is not a smarter metric. It is a more honest label: say what was counted, over which period, in which time zone, and where the count stopped. A number that tells you its own limits can be wrong without misleading you.

Every figure on the home page is now counted from the store's own records over the last 30 days, and labelled for exactly what it is.

• Sales after a chat. Orders placed within seven days after a chat, matched by the shopper's email. We call it exactly that because that is exactly what it is. It doesn't claim the chat caused the sale; some of those shoppers would have bought anyway. It also misses shoppers who chatted without leaving an email, so it undercounts as well as overcounts. It is a count, with its limits stated.
• Conversations, and the share finished without a person stepping in.
• Sales and orders straight from Shopify, in the store's own currency as Shopify reports it, never an assumed euro.
• Daily lines drawn from the real day-by-day figures, flat when the store was flat.
• Satisfaction as the share of thumbs-up from shoppers who rated a chat, and nothing at all when nobody did.
• Demand Radar counts how often shoppers asked for something you don't stock, without a euro figure that would have needed an invented close rate.
• No forecast. The future can't be counted, so we don't draw it.

The same week we fixed two quieter cousins of the problem. The store's currency is now whatever Shopify says it is, refreshed automatically. And the minimum cart value for each discount code is one the merchant sets, in their own currency, rather than a fixed table of 50, 100 and 200 that meant something different in every currency it was shown in.

You don't need to see anyone's code to apply this. Ask these four questions of any figure before you make a decision with it, in any tool, ours included.

• 1 · Can you name the records it was counted from? “Your Shopify orders from the last 30 days” is an answer. “Our model” isn't.
• 2 · Is there a constant inside it? 30%. Five minutes. × 1.2. If there is, the result is mostly that assumption, however convincingly it moves. This is the question borrowed credibility hides, so ask it directly.
• 3 · Does the label claim more than the count? “Sales after a chat” is a count. “Revenue from AI” is a claim about cause, wearing a count's clothes.
• 4 · What does it show when there is no data? An honest number goes blank. A generated one shows 3.0 out of 5, a health score of 70, or “+100%” against an empty previous period.

An estimate isn't always wrong, and some are genuinely useful. A forecast with a stated method and a range can help you plan stock. But a useful estimate says it is an estimate, shows its assumption, and never shares a tile with a measurement as if the two were the same kind of thing.

Black Friday falls on 27 November this year, and many stores freeze their setup well before it. The decisions made in the next few weeks (how much stock, how many people on support, which tools to keep paying for) are the ones most worth making on counted numbers.

So here is our suggestion for the week ahead, whether or not you ever use Lumnya. Open every dashboard you rely on, pick the five numbers you look at most, and run the four questions on each. If one fails, find out what it is really made of before you plan around it. Most vendors will tell you if you ask; the ones who can't are telling you something too.

This is the same principle behind how our assistant talks to your customers, which we wrote about in A prompt is not a contract: the system, not the model, decides what may be said. On the dashboard, the store's records, not a constant, decide what may be shown. Lumnya's specialist agents work from the same records, show their reasoning, and anything they propose that touches money waits for your approval. How approvals work is in our FAQ.

A number you can't count yourself is an opinion with a chart attached.

The same rule applies to what an AI does, not only what it reports: see a check after the action is a log, and, for the screen itself, why the ecommerce dashboard era is over.

• [1] Lumnya source code and change history: the home metrics module (“every number measured, none estimated”) and app changes #294, #295, #296 and #299, 10–12 October 2026, covering the home page, analytics, heatmaps, Demand Radar, exports, settings, currency, discount thresholds and the weekly snapshot. Verified on 19 October 2026.
• [2] J. Becker, N. Rush, E. Barnes, D. Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”, METR, 10 July 2025; paper: arXiv:2507.09089 (accessed 19 Oct 2026).
• [3] Goodhart's law, Wikipedia; the widely quoted wording is M. Strathern's (1997) generalisation of C. Goodhart (1975) (accessed 19 Oct 2026).
• [4] Lumnya, A prompt is not a contract, 18 October 2026.

Lumnya is a team of ecommerce AI agents for Shopify stores, now in private beta. Join the waitlist.