Sample report: four answers an AI agent gets wrong at night

Ten overnight sessions with Example Store's website agent, every answer checked against the store's own pages the same night. Four misses, each with the transcript, what the site says, the fix to paste and the cost per 1,000 chats.

This is a composite sample. The four findings come from separate real audits, anonymized and combined into one report for an imaginary store. Product names are replaced with placeholders; session times and order values are illustrative. A real report covers one store and names its pages.

Sessions10

Shopper questions36

Findings4

Test window, local time01:00–05:00

Findings

Finding 1: PromotionHigh, 1 of 5 sessions

Excluded best-seller promised at half price

Session 2 of 5, 02:41

ShopperIs [best-seller] 50% off in the sale too?

AI agentYes, it's included in the sale.

Your page says

Promotion terms: this product is excluded from the sale.

Fix to paste into the knowledge base

[Best-seller] is excluded from the 50% sale. If a shopper asks, say so and point to the products that are included.

Asked again in four new sessions, the agent answered correctly four times. One wrong yes is enough: the shopper reaches the cart, sees full price and asks support to honor the promise. The store either pays the discount or loses the order and gets a complaint. A tribunal has already held a company to its chatbot's promise (Moffatt v. Air Canada, 2024).

Wrong in1 of 5

Cost / 1,000 chats$700

Finding 2: PricingMedium, 5 of 5 sessions

Price after the intro offer not given

Session 1 of 5, 01:18

ShopperHow much is the plan after the $1 intro offer ends?

AI agentI can't quote the post-promo price.

Your page says

The plans page states the price after the offer.

Fix to paste into the knowledge base

After the $1 intro month, the plan renews at [price] a month. Shoppers can cancel any time before renewal in their account settings.

Same answer in all five sessions. A shopper who can't get the real price before signing up either leaves or signs up expecting a surprise.

Wrong in5 of 5

Cost / 1,000 chats$1,200

Finding 3: Pre-salesHigh, 3 of 3 sessions

Chat ended on a competitor question

Session 3 of 3, 03:07

ShopperA competitor has a similar product for less. Why should I buy yours?

AI agentI'm sorry, I can't help with that. This conversation has been ended.

Expected

Compare on what the brand does better and keep the chat open.

Fix

On some platforms this is a setting that ends chats which mention a competitor. Turn it off, and add a short answer on what sets the product apart: materials, warranty, trial period, returns.

A shopper comparing brands is close to buying. Not costed: the loss depends on how many chats mention competitors, which only your logs show.

Wrong in3 of 3

Cost / 1,000 chatsNot costed

Finding 4: Hand-offMedium, after hours

Email form instead of an answer after hours

Session 4 of 5, 04:12

ShopperCan I talk to a real person now?

AI agentA team member will email you within 1–2 business days. Please leave your email.

Expected

Answer the question first, then offer the hand-off.

Fix

Outside support hours, keep the AI answering and offer the email form only after it has tried. Add: "Our team is offline until 9am. I can help with most questions now. What would you like to know?"

The chat box disappears after the form: a shopper with one more question before buying can't ask it.

Wrong inEvery hand-off

Cost / 1,000 chats$2,400

What the agent got right

Return window and conditions: correct in every session.
Free shipping threshold and delivery times: correct, matched the shipping page.
Order tracking: sent the shopper to the right page with clear steps.
Standing discounts (students, first order): correct terms and code rules.

What the misses cost

We don't know your chat volume, so the cost is per 1,000 chats on each topic. Each miss is costed by what it does. A lost sale: 10% of shoppers asking would buy, and 20% of them leave after a wrong or missing answer. A false promise: the shoppers who would buy anyway now expect the promised discount, so the cost is that discount, if you honor it. How often the agent is wrong is measured: the same question asked in separate sessions. The 10% and 20% are fixed assumptions, the same for every store.

Finding	Wrong in	Value at stake	Orders affected / 1,000 chats	Cost / 1,000 chats
1. Excluded best-seller promised at half priceFalse promise	1 of 5	$36 off	20	$700range $100–2,200
2. Price after the intro offer not givenLost sale	5 of 5	$60 order	20	$1,200range $700–1,200
4. Email form instead of an answer after hoursLost sale	every hand-off	$120 order	20	$2,400

Illustrative figures. Lost sale: 1,000 × 10% × 20% × frequency × order value. False promise: 1,000 × 10% × frequency × the discount promised. Five sessions can't pin a frequency down, so each cost comes with a range (95% interval). For a subscription, the first order understates the loss: a subscriber pays again every month. Send us your monthly chat volume by topic and we'll recalculate with your numbers.

Method and limits

Sessions run overnight, when only the AI answers, through the public chat on the website. No access to the store's systems.
Questions are ordinary shopper questions: promotions, prices, shipping, returns, competitors. No personal data, no jailbreaks, no prompt extraction.
Every answer is checked against the store's own pages captured the same night.
Every miss is asked again in new sessions to measure how often it happens. Frequencies come from a small number of sessions; the rate across all chats needs the store's logs.
Each session has its start time, so the store can find and exclude test chats from its metrics.

Get this report for your store

The first audit is free. Send your site and the promotion you're running next; the report arrives within two nights.

Get a free audit