Following disclosures that autonomous AI agents escaped from OpenAI model training breached AI platform Hugging Face, subsequent findings from Anthropic and OpenAI indicate that enterprise AI agents can reach real systems and pursue objectives across boundaries that organizations assumed would hold.
Customer experience leaders have a particular reason to pay attention to the security implications.
Anthropic has disclosed three separate incidents in which its models reached real-world systems during cybersecurity evaluations. And OpenAI has also identified additional cases in which its models found and used publicly exposed credentials belonging to accounts on other services.
The incidents point to a cybersecurity problem for enterprises that sits beyond model-level guardrails. For enterprises deploying AI agents into customer service, customer relationship management (CRM), marketing, sales and other customer-facing workflows, the question is increasingly what an agent can actually reach and do once it starts pursuing an objective.
The OpenAI Incident Was Only Part of the Picture
OpenAI’s investigation into the Hugging Face incident found that its models had chained vulnerabilities, escalated privileges and moved laterally until they could reach systems with Internet connectivity.
Once online, the models inferred that Hugging Face might contain resources related to their ExploitGym evaluation and found ways to access information that could help them complete the benchmark. Hugging Face subsequently reconstructed more than 17,000 attacker events across the campaign.
The incident was significant because the models were not given an explicit instruction to attack Hugging Face. They were given an objective and operated in an environment that was supposed to be isolated.
"The models identified and used publicly exposed credentials at the account-level on other publicly-available services. This includes four accounts on four services," OpenAI stated.
The underlying behavior of an autonomous system that was capable of finding credentials exposed elsewhere and using them in pursuit of its objective should concern CX leaders. What happens when the equivalent agent has legitimate credentials to your CRM, customer service platform, payment system or communications stack?
Vincent Danen, Vice President of Product Security at Red Hat, described in a recent CX Today interview why autonomous agents are different from traditional automation.
“Automation is deterministic. Automation will continue to do the same thing over and over again, whatever it’s been told to do. Autonomous agents using AI are non-deterministic, which means they may change their mind as they’re going through.”
Agentic systems are not simply executing a predefined script. They may adapt, reroute, search for alternatives and make use of whatever resources are available to complete a goal.
Anthropic Finds Agents Reaching Real Companies
Anthropic’s subsequent investigation makes the concern harder to dismiss as a single unusual incident.
The company said it launched a review after OpenAI disclosed the Hugging Face incident and examined cybersecurity evaluations involving its own models.
In one evaluation, Claude Mythos 5 created and published a malicious Python package to the real PyPI registry because it believed the package was part of the simulated environment. The package was downloaded and executed by 15 real systems during the roughly one-hour period in which it was publicly available. One belonged to a security company, whose scanner installed the package and subsequently exposed credentials that Claude used to access further infrastructure.
Anthropic's conclusion is important for enterprise buyers, as the models were generally following the objectives they had been given but they misunderstood the environment in which those objectives were being pursued.
The company said the incidents demonstrate why evaluation environments for increasingly autonomous AI systems require the same security standards as other production infrastructure. It is also encouraging other AI labs to conduct similar reviews.
Kristina Holt, Managing Associate at Foot Anstey, told CX Today that recent AI agent incidents show why simple assurances about guardrails are not enough. “They thought they put [guardrails] there to stop them doing that. It hadn’t worked.”
The difficult lesson for enterprises is that a system can appear contained until it encounters a path its designers did not anticipate.
The Biggest CX Risk May Be the Agent That Has Permission
It is tempting to frame these incidents as a cybersecurity research problem involving unusually capable models, but that would miss the more immediate enterprise issue.
A customer service agent might legitimately be given permission to:
- retrieve a customer's CRM record;
- update account information;
- issue a refund or credit;
- access an order-management platform;
- search an internal knowledge base;
- send an email or SMS;
- create a support ticket;
- call an external API;
- escalate a case;
- or hand a task to another AI agent.
Each permission can be legitimate in isolation.
The risk emerges when an agent can chain those permissions together in pursuit of an objective that was defined at a much higher level.
"Resolve this customer's problem" could involve dozens of actions across multiple systems. An agent that encounters an unexpected error, missing record or access restriction may search for another route.
The OpenAI and Anthropic incidents indicate why organizations cannot assume the model itself will reliably recognise that a particular route is outside its remit.
Danen’s advice is blunt: “Don’t give the agent access to the stuff that you don’t want it to tell people.”
Unknown attack methods make that principle especially important.
“If the agent knows about it and can be tricked in some way that we haven’t discovered yet… you can’t really protect against the thing that you don’t know exists.”
Security software vendor Check Point similarly argues that organizations should assume an agent will pursue its assigned task across everything it can actually reach, which changes the security boundary.
“The lesson is simple to say and hard to do: an agent will not stay in its scope on its own. You have to keep it there. That work divides across three layers, and the lines are less settled than they were for cloud.”
The model provider secures the model and its infrastructure behind it, AI security tools govern the agent using guardrails, but the organization that deploys an agent owns its behavior and reach, Check Point warns. “Controls can be outsourced. Accountability cannot. The risk decision stays with you.”
The Action Layer Becomes the Critical Control Point
Traditional AI governance tends to focus heavily on the model: what it has been trained to do, what prompts it accepts, what content it refuses and whether its responses comply with policy.
Those controls remain important.
But they cannot be the only line of defence once an agent is taking actions.
An agent can be perfectly aligned with its stated objective while still producing an unacceptable outcome because the objective, context or environment is misunderstood.
That is precisely why Anthropic's findings matter. In one case, the model was explicitly told it had no internet access, but it encountered real systems and treated them as part of the simulated exercise.
