First published 24 June 2026; rewritten and expanded 19 October 2026. There are two futures of AI being sold to store owners, and both are wrong. In the first, an AI runs your store while you sleep, and every mistake it makes is one you discover afterwards, in your revenue, with no explanation attached. In the second, the AI only ever suggests, and every output lands in your lap as one more thing to read, edit and do yourself. The usual answer to both is “keep a human in the loop”. This piece is about why that answer, on its own, is not enough, and what has to be true for a human's yes to actually protect anything.
- A human in the loop protects you only if the human is actually checking. Decades of research on automation say people stop checking reliable-looking systems, and that training doesn't fix it.
- So a gate has two failure modes: it can be missing, or it can be asked so often that the yes becomes a reflex.
- The answer is to gate by consequence, not by default: how hard the action is to undo, and whether it reaches a customer or money.
- Some actions should never run without a yes, at any setting. In Lumnya those are refunds, return decisions, emails to customers and live-chat messages.
- A yes means something only if the preview is falsifiable: the exact change, checkable in two seconds. “Update pricing?” isn't a question you can answer. “520.00 → 572.00” is.
Autonomy without accountability is a liability. Autonomy with a receipt is a colleague.
The human in the loop stops looking
Human factors researchers have studied what happens when people supervise automated systems for decades, in cockpits, control rooms and clinics. A widely cited review by Raja Parasuraman and Dietrich Manzey, “Complacency and Bias in Human Use of Automation” (Human Factors, 2010), pulled the findings together. Two of them matter for anyone putting an AI in charge of part of a store.
- Complacency. When a system is usually right, people monitor it less than they should, especially when they have other things competing for their attention. Expertise and practice don't remove the effect.
- Automation bias. People over-rely on an automated recommendation, both missing problems the system didn't flag and following advice they would have rejected had they worked it out themselves. Training and instructions don't reliably remove it either.
Now picture a store owner with an AI that asks for approval forty times a day, and is right thirty-nine of them. By the end of the second week, the approve button is a reflex. The gate is still there, technically. It has stopped doing its job. A gate that asks about everything protects you from nothing, because the one request that deserved attention arrives looking exactly like the thirty-nine that didn't.
The opposite failure is better known. In July 2025, during a public trial, an AI coding agent on Replit deleted a live production database, despite an instruction not to make changes without approval, and then misreported what had happened. The instruction was in the conversation. Nothing outside the model enforced it. We wrote about that difference, between a rule the model is asked to follow and a rule the server enforces, in A prompt is not a contract.
This is no longer only an academic concern. The EU's AI Act writes it into law for high-risk systems: Article 14 requires that people overseeing such a system can “remain aware of the possible tendency of automatically relying or over-relying on the output”, and can decide not to use it, override it, or stop it. A store's chat assistant is not a high-risk system under the Act. But when legislators name automation bias explicitly, it is a fair sign that “a human approves it” is not, by itself, a safety argument.
Gate by consequence, not by default
If asking about everything doesn't work, and asking about nothing is reckless, the question becomes which actions deserve a yes. We sort them by two properties, and only two.
- How hard is it to undo? A price change can be reversed in seconds. An email to a customer can't be unsent. A refund, once it reaches the card, is gone.
- Does it reach a customer or money? Changing an internal note reaches nobody. Publishing an FAQ answer reaches every shopper who asks. Issuing a discount reaches your margin.
Figure 1 · Where the gate goes
The harder an action is to undo, and the further it reaches, the stronger the gate
Actions Lumnya's assistant can take, placed by how hard they are to undo and whether they reach a customer or money.
↑ Reaches customers or money
Asks first · can earn a standing yes
Never automatic, at any setting
Low stakes
Asks every time
Harder to undo →
The top-right corner is the important one. Some actions are both irreversible and customer-facing, and for those a standing approval would be exactly the reflex the research warns about. So in Lumnya, refunds, return decisions, emails to customers and live-chat messages can never run without your yes, at any setting, and so can any change to the autonomy settings themselves. That rule lives on the server, not in the model's instructions: there is no configuration, prompt or approval history that turns it off.
Everything else asks first by default. The difference is what happens over time. For reversible actions, it is reasonable to let a track record count. We have built that step and keep it off by default: after you have confirmed the same kind of action three times, Lumnya can offer a standing yes for that kind of action, inside caps on the number of actions per day, the amount per action and the amount per month, with quiet hours, and you can pause it at any time. It never applies to the top-right corner.
Figure 2 · The rules, action by action
What asks, what can earn a standing yes, and what never runs alone
Lumnya's assistant actions against the three gate rules, as implemented.
| Asks first by default | Can earn a standing yes | Never automatic | |
|---|---|---|---|
| Refund | yes | no | yes |
| Return decision | yes | no | yes |
| Email to a customer | yes | no | yes |
| Live-chat message | yes | no | yes |
| Stock count overwrite | yes | no | yes |
| Price change | yes | yes | no |
| Discount code | yes | yes | no |
| FAQ answer | yes | yes | no |
yesnon/a
A yes is only as good as the preview
The second half of the problem is what the question looks like. A vague approval request is worse than none, because it gives the appearance of oversight with none of the substance. You can't check “Update pricing?” against anything, so you approve it on trust, which is exactly the automation bias the research describes.
Figure 3 · Two ways to ask for the same yes
A falsifiable preview turns trust into a two-second comparison
Four approval requests, as a vague gate asks them and as a real gate shows them.
What a vague gate asks
Preview
What a real gate shows
“Update pricing?”
“Oxygen Pro, Medium: 520.00 → 572.00. Can be undone.”
“Handle this return?”
“Approve the return for #1047 and refund 89.00 to the original card. This can't be undone.”
“Send follow-up emails?”
“Email 1 customer: “Your replacement ships Monday.” Full text below.”
“Apply a discount?”
“Create AUTUMN10: 10% off carts over 60, ends Sunday.”
The rule we hold ourselves to: a preview must let you catch a mistake without opening anything else. That means the exact before and after, the specific customer or order, the amount, and whether it can be undone. In Lumnya, a yes also applies only to the exact proposal you were shown: if anything about the action changes, it asks again, and an unanswered proposal expires after fifteen minutes instead of waiting to be approved later out of context.
Earning a standing yes
The idea behind standing approvals is the one you would use with a new hire. Nobody gives a new employee the company card on day one, and nobody makes a five-year veteran ask permission to send a routine email. Trust is granted per kind of task, on the strength of a record you have watched.
The difference with software is that the record can be exact. After three confirmations of the same kind of reversible action, Lumnya can offer to stop asking for that kind of action, and only that kind. The standing yes carries limits you can see: how many actions a day, how much per action, how much per month, and hours in which it never acts. Every action taken under it is journaled as such, so you can tell at a glance what you approved and what ran on the standing yes. Revoking it is a single instruction, and it is off unless you turn it on.
The receipt: what the log records, and what it doesn't yet
Approvals cover the moment of action. The log covers every moment after: when a price looks wrong next Tuesday, “why did it do that?” should have a concrete answer. Here is exactly what Lumnya records today, including the gaps.
| What is recorded | Kept | Today |
|---|---|---|
| Every action an agent takes: which agent, what it did, to what, the result or the error, and when | Without a time limit | ✓ |
| How each change was made: immediately, after your confirmation, or on a standing approval | 30 days | ✓ |
| What it takes to undo the change, so it can be reversed | 30 days | ✓ |
| What happened afterwards, where it can be measured: orders using a discount code, changes in how often a trigger fires, changes in FAQ deflection | With the action | ✓ |
| Which person approved it, for stores where several people can approve | — | Not yet |
| The reasoning behind each action as its own field in the log | — | Not yet |
The last two rows matter. An earlier version of this article said the log records who approved an action and its reasoning. It doesn't yet, and we have corrected the article rather than leave it standing. For a single owner the first gap is small, since the approver is you. For a team it is real, and we will say so until it is closed.
How to audit human-in-the-loop AI in any tool
You can test this in an afternoon, with any vendor, without seeing their code. Ask these and watch for mechanisms rather than adjectives.
- “Which actions can never run without my approval, and where is that enforced?” “In the prompt” is a preference. “On the server, per action” is a rule.
- “Show me a preview for a refund.” If it doesn't show the order, the amount and whether it can be undone, your yes will be a guess.
- “What happens if I approve, and the action changes before it runs?” The right answer is that it asks again.
- “How many approvals will I see in a normal week?” If the honest answer is hundreds, you will stop reading them, whatever the vendor intends.
- “How do I undo the last thing it did?” If the answer involves support tickets, the gate is the only protection you have.
- “What does the log record, and what doesn't it?” A vendor who can name the gaps in their own log is one whose log you can trust.
What we refuse to ship
- Silent changes. If it changes your store, you saw the exact change first, or you chose a standing yes for that kind of change.
- Default autonomy. Asking first is the factory setting, always.
- Autonomy for irreversible, customer-facing actions. Refunds, return decisions, customer emails and live-chat messages ask every time.
- Vague approvals. If a preview can't be checked in two seconds, it isn't a preview.
- Trust theatre. No borrowed badges and no claims we can't back. Our security page lists what is actually true.
The industry will keep promising fully autonomous everything, because it demos well. We will keep building the gate, and keep it asking only where it matters, because stores run on it. How approvals work in practice is in our FAQ.
Two pieces that follow from this one: a check after the action is a log, with what developers actually wire around their own agents, and the rule for what an assistant may claim about stock and deadlines.
Frequently asked questions
What is confirm-gated AI?
An AI system that proposes an exact change and waits for a person's approval before acting, then records what it did. The design question is which actions need the approval: gating everything causes approval fatigue; gating nothing is reckless.
Is human-in-the-loop AI safe?
Only if the human is really checking. Research on automation (Parasuraman & Manzey, 2010) finds people monitor reliable systems less than they should and over-rely on their recommendations, and training doesn't remove the effect. Gates work best when they are reserved for consequential actions and show a precise, checkable preview.
Which actions should an AI never take on its own in a store?
Actions that are hard to undo and reach a customer or money. In Lumnya, refunds, return decisions, emails to customers and live-chat messages always wait for the owner's approval, at every setting, and that rule is enforced on the server.
Can Lumnya act without asking?
Not by default: every change to your store asks first with a preview. A standing approval for reversible kinds of action, within daily and per-action caps, exists and is off unless enabled. It never applies to refunds, return decisions, customer emails or live-chat messages.
Sources
- [1] R. Parasuraman, D. H. Manzey, “Complacency and Bias in Human Use of Automation: An Attentional Integration”, Human Factors 52(3), 381–410, 2010 (accessed 19 Oct 2026).
- [2] TechTarget, “Replit AI agent snafu: shot across the bow for vibe coding”, July 2025 (accessed 19 Oct 2026).
- [3] Regulation (EU) 2024/1689 (AI Act), Article 14: Human oversight (accessed 19 Oct 2026).
- [4] Lumnya product behaviour described here was verified in Lumnya's source code on 19 October 2026: the confirm default, the never-automatic action list, standing approvals and caps, the 15-minute proposal expiry, the activity log and the action journal.
Lumnya is a team of ecommerce AI agents for Shopify stores, now in private beta. Join the waitlist.
Comments
…Every comment is read and approved by a human before it appears — the same confirm-gated rule the product runs on.
Loading comments…
Filed under Product
All posts