A Cyber News experiment has once again exposed the cracks in an AI chatbot, this time Snapchat’s My AI, offering a stark reminder for companies rushing to put artificial intelligence in the customer experience driver’s seat.
Cyber News researchers recently tested Snapchat’s AI chatbot, which is used by over 900 million people worldwide, with some creative prompting that framed requests as storytelling exercises to trick the bot into sharing instructions for making improvised explosive devices, like Molotov cocktails.
While Snapchat’s safeguards block direct queries about weapons, the chatbot recited historical “how-tos” under the guise of a narrative when the team prompted it to tell a story about the Winter War between Finland and the Soviet Union and include details about how incendiary devices were reportedly made at the time.
This instance raises concerns about what other dangerous content could slip through, especially to younger users.
The Cyber News team explained:
“While the bot may never directly provide instructions on how to build improvised weapons, it will tell you a realistic and detailed story of how improvised weapons used to be built without any hesitation. This raises concerns about dangerous AI information availability for minors.”
The researchers notified Snapchat, but the vulnerability wasn’t patched immediately.
Snapchat says My AI is trained on a broad range of texts and built with safety features it claims are “unique to Snapchat.” The training process “was designed to avoid amplifying harmful or inaccurate information, and the model was also fine-tuned to reduce biases in language and to prioritize factual information — though it may not always be successful,” Snapchat’s website states.
The experiment indicates that Snapchat’s claimed guardrails might not be as safe as they seem.
“Of course, no one’s rushing to Snapchat for lessons in destruction. But the experiment shows just how easily an AI can be pushed past the limits of what it was meant to do,” according to Cybernews.
It also underscores how easily AI systems can be pushed beyond their ethical or operational limits.
Snapchat isn’t alone. Meta, Lenovo, and other platforms have all faced similar AI “jailbreaks,” where chatbots are manipulated into providing unsafe or sensitive information.
Back in August 2025, Cyber News researchers tricked Lenovo’s AI chatbot Lena into exposing sensitive company data using a 400-character prompt.
This was enough to manipulate the bot into running unauthorized scripts on corporate machines, spill active session cookies, and sift through past conversations. Attackers can abuse such XSS vulnerabilities as a direct pathway into a company’s customer support platform.
Lenovo patched the flaw quickly, but the vulnerability shows how companies deploying AI chatbots can be exposed to massive data breaches that compromise customer trust.
These aren’t just abstract security risks; they have implications for customer experience teams relying on AI assistants to provide accurate information.
Another recent example is AI startup Anysphere and its AI-powered coding assistant Cursor’s chatbot, Sam, which responded to a customer query with a company policy that did not exist.
Several Reddit users shared their frustrations publicly and stated that they were cancelling their subscriptions to Cursor. By the time Anysphere responded three hours later, the story of a chatbot inventing a policy had already gone viral.
The Cursor incident illustrates the same fundamental problem as the Snapchat exploit. Chatbots can confidently present false, or even harmful, information as fact. This raises a crucial question for CX teams—how much can you trust the AI handling your customer interactions?
AI can speed up support and handle repetitive tasks, but companies need to combine automation with oversight, routinely testing AI behavior and keeping humans in the loop for high-stakes or policy-sensitive interactions.
The question isn’t whether AI will make mistakes, it’s how prepared you are to catch them before your customers do.
