Today, the growing use of AI in CX has created a risk of hallucination, delivering confident but inaccurate information to customers at scale.
The issue gained renewed attention after law firm Sullivan & Cromwell acknowledged AI-generated errors in a federal court filing, demonstrating that even highly reviewed environments are vulnerable to convincing AI mistakes.
For CX leaders, customer-facing AI now requires further attention, as a single error can quickly become a widespread trust, compliance, and reputational problem.
Jackie Swanson, Managing Partner at Gartner Consulting, told CX Today that CX AI hallucinations go largely unreviewed, enabling AI systems to confidently deliver incorrect information that looks indistinguishable from correct answers.
“Court filings get scrutinized. CX interactions do not. The same model that confidently invented a case citation in front of a federal judge is confidently inventing return policies, warranty terms, and product specifications inside enterprise chatbots every day,” she explained.
“The hardest part about CX hallucinations is that they are invisible by design. A model gives a wrong answer with the same tone and the same fluency as a right one.”
In April, the Wall Street law firm Sullivan & Cromwell admitted to submitting a bankruptcy court filing containing AI-generated errors.
This included numerous inaccuracies such as fabricated citations, incorrect quotations, and references to legal authorities that either did not exist or had been misrepresented.
Furthermore, reports indicate that these errors spanned dozens of citations and included passages of other real cases, illustrating how confidently presented AI outputs can appear credible even when they are incorrect.
And whilst the incident was caught early, these highlight why AI hallucinations can be particularly dangerous in CX environments, where many AI-generated responses reach customers immediately, often without any human review.
Speaking with CX Today, Sushil Kumar, CEO at Cyara, emphasized how the court case demonstrated the limits of human oversight, showing that even highly skilled manual review cannot effectively scale to monitor the volume of outputs produced by modern AI systems.
"Even at a highly respected law firm, with intelligent lawyers and human review in place, AI-generated errors still slipped through,” he noted.
“Humans aren't built to audit machines at scale. A single agent trying to review a million conversations a day simply isn't possible. And when AI makes a mistake, it doesn't announce itself.”
Unlike a legal filing, CX systems can distribute inaccurate information simultaneously across channels, with customers likely acting on those responses and creating highly reputational consequences for the organization.
Unfortunately, plausibility is often what makes hallucinations so difficult to detect, and in CX, the same dynamic as in a courtroom can allow errors to spread even faster.
“There's no flashing red light, no obvious signal that something is wrong,” he pointed out.
“The response sounds fluent, polished, and confident, even when it's inaccurate, and that's the trap CX leaders are falling into.”
As a result, installing robust monitoring systems, escalation workflows, and human oversight therefore becomes essential safeguards against AI-generated misinformation reaching customers at scale.
Why AI Hallucinations Are Potentially More Dangerous in CX
In CX, AI hallucinations present a unique risk because customer-facing systems operate at an enormous scale and often without human review.
In contrast to a courtroom, an AI assistant with inaccurate information will spread this across thousands or even millions of interactions before the issue is identified and corrected.
In fact, this challenge is rooted in the reality of customer behavior, with their frequent unpredictable nature during interactions often falling outside of clean, predictable paths.
"No customer behaves like a test script. Real customers interrupt, change their minds, get emotional, and call customer service lines at 11pm angry about a bill,” Kumar highlights.
“Even if a bot handles 95% of interactions well, the remaining 5% can become the moments that matter most.”
This seemingly minor issue could represent thousands or millions of customers receiving incorrect information, failing to resolve their issues, or leaving interactions more frustrated.
Furthermore, customers often judge AI mistakes more harshly than human ones, with Cyara’s research revealing 61% of customers reporting that bot failures are more frustrating than human errors.
As a result, even relatively infrequent hallucinations can have outsized effects on customer trust and satisfaction.
Is AI Quietly Rewriting the Customer Relationship?
Furthermore, the business risks posed by AI hallucinations can also alter customer expectations and perceptions of a brand.
This trust dynamic between brand and customer makes AI-generated misinformation particularly dangerous, as customers are not able to accurately distinguish between human-provided and AI-provided information.
Customers don't have a relationship with the model provider, they have a relationship with the brand,” Kumar stated.
“If an AI assistant gives the wrong answer, the customer won't be angry at the model, they'll be angry with the company. The enterprise chooses the model, the deployment, and the testing, so it is ultimately responsible for the outcomes.”
When responsibility rests with the enterprise deploying the technology, organizations cannot deflect problem ownership when AI-generated errors affect customers.