A chatbot making a mistake isn’t exactly a rarity, but one deciding to create a new company policy is a different kind of AI inadequacy.
Unfortunately for Cursor – the AI-powered software coding assistant from AI startup Anysphere – this is exactly what its chatbot did in response to a recent customer service query.
In a post shared on Reddit, a Cursor user (BrokenToasterOven) explained that they had contacted the company’s customer support department having experienced issues with being logged out of the platform when switching between machines.
In response, an AI chatbot named ‘Sam’ stated that “Cursor is designed to work with one device per subscription as a core security feature.”
Unbeknownst to the user, this policy did not exist.
In response to BrokenToasterOven’s post, many Reddit users publicly shared their frustrations and announced that they were cancelling subscriptions to Cursor.
Customers described the fake policy as “asinine,” “terrible,” and “dumb,” with one claiming that it was “ABSOLUTELY UNACCEPTABLE.
I have ten machines that I run at once, and I never use slow requests. I don’t want ten different subscriptions. I’m cancelling today, that’s f*****g it.
BrokenToasterOven also joined the chorus of dissatisfied users, stating:
“I’m out, that’s it, they have taken too much, asked too much, and given us too little. We are getting core features taken away daily.
“Goodbye, gang. I suggest getting off this ship while you can.”
In response, a Cursor representative did reply to the Reddit thread, explaining that there is no such policy in place and that users are “free to use Cursor on multiple machines.
“Unfortunately, this is an incorrect response from a front-line AI support bot. We did roll out a change to improve the security of sessions, and we’re investigating to see if it caused any problems with session invalidation.”
However, the company’s response did not come until three hours later, by which time the story of a chatbot inventing a policy was well on its way to going viral.
While there is no telling how much this may have cost Cursor in cancelled subscriptions, it is the reputational damage that may prove to be the most costly in the long term.
But how exactly did Cursor’s chatbot end up making such a blunder?
AI Hallucinations Are a Real CX Concern
Those of you who are well-versed in the world of AI will be familiar with the term “hallucination.”
Hallucinations refer to instances where AI tools confidently generate false information instead of acknowledging uncertainty.
While major AI firms are committing time and resources to eradicating hallucinations from their offerings, Cursor’s mishap is the latest in a number of high-profile incidents that have occurred over the past 18 months.
Back in January, a Virgin Money customer was inexplicably reprimanded by the company’s chatbot for using the word ‘Virgin’ during a customer service inquiry.
“Please don’t use words like that. I won’t be able to continue our chat if you use this language,” the bot replied.
While the interaction proved to be a hit on social media, with hundreds of people seeing the funny side, it is another instance of a glaring error caused by a hallucination.
In an example of legitimately inappropriate language, DPD made headlines for all the wrong reasons last year when a frustrated customer managed to convince one of its chatbots to swear and write a poem about how terrible the company is.
Away from some of the more amusing situations that have been brought about by chatbot hallucinations, there have also been those with legal implications.
In February of last year, a small claims court ruled that Air Canada must compensate a customer misled by a chatbot into overpaying for a flight.
Jake Moffatt, who was traveling for a bereavement, was incorrectly told he could claim a refund after buying full-price tickets. He paid over $1,600 instead of the bereavement fare of around $760.
Elsewhere, New York City’s generative AI (GenAI) chatbot, ‘MyCity,’ recently received heavy criticism for providing small business owners with illegal advice and misrepresenting local policies.
These more serious byproducts of AI hallucinations were discussed in a Gartner report from last year.
In a very troubling prediction, Gartner stated that “by 2027, a company’s generative AI chatbot will directly lead to the death of a customer from bad information it provides.”
Although the recent incident with Cursor does not appear to have any more serious ramifications than a loss of customers and some reputational damage, it still serves as a warning to any organizations looking to deploy AI chatbots.
There are irrefutable benefits to these automated agents, but when not implemented and tested correctly, they can also be detrimental to a company’s customer service and experience services.
