An AI that answers customers without a human check can state a policy, a refund window or a rule that does not exist. Customers believe it, act on it, and the company is held to it or loses the customer.
The model fills gaps with plausible text when the answer is not in its sources, and nothing stops the reply from reaching the customer. In the Cursor case, a co-founder said AI-assisted replies were the first filter for email support.
Let the agent draft only from written sources, and show the source under each draft. Block any promise of money, terms or features not in those sources. Route refunds, legal, security and upset customers to a person. Keep a person on every send. Do not sign bot replies with a made-up human name. When a wrong answer gets out, correct it publicly and fast.
Users report Cursor's support bot described a logout bug as expected under a policy that did not exist. A co-founder apologised in the thread. Other commenters report invented answers from other companies' support bots.
Fortune reports the invented policy spread online, some users cancelled, and critics faulted the lack of disclosure that the agent was a bot.
Fox Business reports the airline's chatbot gave wrong bereavement refund advice and a tribunal ordered the airline to pay the customer.
The Markup found a city chatbot gave wrong legal answers and gave different staff opposite answers to the same question.