Following disclosures that autonomous AI agents escaped from OpenAI model training breached AI platform Hugging Face, subsequent findings from Anthropic and OpenAI indicate that enterprise AI agents can reach real systems and pursue objectives across boundaries that organizations assumed would hold.
Customer experience leaders have a particular reason to pay attention to the security implications.
Anthropic has disclosed three separate incidents in which its models reached real-world systems during cybersecurity evaluations. And OpenAI has also identified additional cases in which its models found and used publicly exposed credentials belonging to accounts on other services.
The incidents point to a cybersecurity problem for enterprises that sits beyond model-level guardrails. For enterprises deploying AI agents into customer service, customer relationship management (CRM), marketing, sales and other customer-facing workflows, the question is increasingly what an agent can actually reach and do once it starts pursuing an objective.
The OpenAI Incident Was Only Part of the Picture
OpenAI’s investigation into the Hugging Face incident found that its models had chained vulnerabilities, escalated privileges and moved laterally until they could reach systems with Internet connectivity.
Once online, the models inferred that Hugging Face might contain resources related to their ExploitGym evaluation and found ways to access information that could help them complete the benchmark. Hugging Face subsequently reconstructed more than 17,000 attacker events across the campaign.
The incident was significant because the models were not given an explicit instruction to attack Hugging Face. They were given an objective and operated in an environment that was supposed to be isolated.
“The models identified and used publicly exposed credentials at the account-level on other publicly-available services. This includes four accounts on four services,” OpenAI stated.
The underlying behavior of an autonomous system that was capable of finding credentials exposed elsewhere and using them in pursuit of its objective should concern CX leaders. What happens when the equivalent agent has legitimate credentials to your CRM, customer service platform, payment system or communications stack?
Vincent Danen, Vice President of Product Security at Red Hat, described in a recent CX Today interview why autonomous agents are different from traditional automation.
“Automation is deterministic. Automation will continue to do the same thing over and over again, whatever it’s been told to do. Autonomous agents using AI are non-deterministic, which means they may change their mind as they’re going through.”
Agentic systems are not simply executing a predefined script. They may adapt, reroute, search for alternatives and make use of whatever resources are available to complete a goal.
Anthropic Finds Agents Reaching Real Companies
Anthropic’s subsequent investigation makes the concern harder to dismiss as a single unusual incident.
The company said it launched a review after OpenAI disclosed the Hugging Face incident and examined cybersecurity evaluations involving its own models.
In one evaluation, Claude Mythos 5 created and published a malicious Python package to the real PyPI registry because it believed the package was part of the simulated environment. The package was downloaded and executed by 15 real systems during the roughly one-hour period in which it was publicly available. One belonged to a security company, whose scanner installed the package and subsequently exposed credentials that Claude used to access further infrastructure.
Anthropic’s conclusion is important for enterprise buyers, as the models were generally following the objectives they had been given but they misunderstood the environment in which those objectives were being pursued.
The company said the incidents demonstrate why evaluation environments for increasingly autonomous AI systems require the same security standards as other production infrastructure. It is also encouraging other AI labs to conduct similar reviews.
Kristina Holt, Managing Associate at Foot Anstey, told CX Today that recent AI agent incidents show why simple assurances about guardrails are not enough. “They thought they put [guardrails] there to stop them doing that. It hadn’t worked.”
The difficult lesson for enterprises is that a system can appear contained until it encounters a path its designers did not anticipate.
The Biggest CX Risk May Be the Agent That Has Permission
It is tempting to frame these incidents as a cybersecurity research problem involving unusually capable models, but that would miss the more immediate enterprise issue.
A customer service agent might legitimately be given permission to:
- retrieve a customer’s CRM record;
- update account information;
- issue a refund or credit;
- access an order-management platform;
- search an internal knowledge base;
- send an email or SMS;
- create a support ticket;
- call an external API;
- escalate a case;
- or hand a task to another AI agent.
Each permission can be legitimate in isolation.
The risk emerges when an agent can chain those permissions together in pursuit of an objective that was defined at a much higher level.
“Resolve this customer’s problem” could involve dozens of actions across multiple systems. An agent that encounters an unexpected error, missing record or access restriction may search for another route.
The OpenAI and Anthropic incidents indicate why organizations cannot assume the model itself will reliably recognise that a particular route is outside its remit.
Danen’s advice is blunt: “Don’t give the agent access to the stuff that you don’t want it to tell people.”
Unknown attack methods make that principle especially important.
“If the agent knows about it and can be tricked in some way that we haven’t discovered yet… you can’t really protect against the thing that you don’t know exists.”
Security software vendor Check Point similarly argues that organizations should assume an agent will pursue its assigned task across everything it can actually reach, which changes the security boundary.
“The lesson is simple to say and hard to do: an agent will not stay in its scope on its own. You have to keep it there. That work divides across three layers, and the lines are less settled than they were for cloud.”
The model provider secures the model and its infrastructure behind it, AI security tools govern the agent using guardrails, but the organization that deploys an agent owns its behavior and reach, Check Point warns. “Controls can be outsourced. Accountability cannot. The risk decision stays with you.”
The Action Layer Becomes the Critical Control Point
Traditional AI governance tends to focus heavily on the model: what it has been trained to do, what prompts it accepts, what content it refuses and whether its responses comply with policy.
Those controls remain important.
But they cannot be the only line of defence once an agent is taking actions.
An agent can be perfectly aligned with its stated objective while still producing an unacceptable outcome because the objective, context or environment is misunderstood.
That is precisely why Anthropic’s findings matter. In one case, the model was explicitly told it had no internet access, but it encountered real systems and treated them as part of the simulated exercise.
For enterprise CX, the equivalent failure could be an agent interpreting an unexpected API response, customer record or third-party service as something it is authorised to act upon.
Holt said the question for organizations using these tools should not simply be whether the technology can perform a task, but whether the organization has control over what it is doing.
The resulting controls need to sit outside the model, so that each agent is treated as an identity with defined privileges, rather than as a software feature with generic access.
It means controlling:
- What can the agent access? CRM, customer data, payment systems, knowledge bases and SaaS applications should each have explicit access policies.
- What can the agent do? Read, write, delete, approve, refund, send and escalate should be treated as distinct permissions.
- Where can it connect? Outbound internet access and API calls should be tightly controlled rather than inherited from the underlying infrastructure.
- How much can it do? Transaction limits, rate limits and approval thresholds can constrain the potential impact of unexpected behaviour.
- What happens when behavior changes? An unusual sequence of API calls, privilege escalation attempts or access to unrelated systems should trigger intervention rather than simply being logged for later review.
Credentials Are Becoming an Agent-Security Problem
The newly disclosed OpenAI findings also highlight a familiar enterprise weakness: exposed credentials.
AI agents are exceptionally well positioned to discover secrets that humans might overlook. A credential buried in a public repository, exposed configuration file, debugging endpoint or third-party service can become an actionable route for an autonomous system capable of searching continuously and correlating information across sources.
This makes credential hygiene part of the AI agent security conversation.
Long-lived, highly privileged credentials are particularly problematic when attached to autonomous systems.
An agent should have its own identity and narrowly scoped permissions, with credentials that can be revoked quickly if its behaviour changes.
The principle should resemble zero trust applied to an employee or application, but with an additional requirement that the system needs to assume the identity itself may act unexpectedly.
Holt noted that permissions are already difficult for many organizations to manage, and autonomous agents increase the stakes.
“We know people are bad at locking down permissions… and it’s administratively very difficult as well.”
Agents taking action create a higher risk profile than systems that only retrieve or summarize information, Holt added.
That is especially the case for CX environments where agents may be connected to CRM data, refund workflows, messaging tools, identity systems and payment-related processes.
Evaluation Environments Deserve Enterprise-Level Security
There is also a lesson here for companies building their own agents.
Anthropic found that simple environmental assumptions contributed to its incidents. OpenAI’s models similarly found a route out of an environment that was intended to be contained.
Enterprises should treat an agent evaluation environment as a potential attack surface.
As testing an agent with production-like data, credentials, APIs or integrations can create a pathway into real infrastructure, it is essential to separate test and production identities, remove unnecessary Internet access, use synthetic customer data where possible, restrict outbound connections and monitor agent activity continuously.
It also means testing the agent against failure scenarios rather than evaluating only whether it completes its intended workflow.
Danen said organizations need to understand the surrounding technologies well enough to reduce the possible blast radius when agents behave unexpectedly.
“Being well versed in those different technologies and keeping abreast of them will ensure your blast radius will be a little bit less if these autonomous agents decide to go off and do things on their own that you didn’t really design or want them to do.”
CX Leaders Need a New Definition of “Contained”
The OpenAI, Hugging Face and Anthropic disclosures demonstrate that containment cannot be treated as a property of the model alone.
A model can have safety controls, an agent can have a system prompt and a test environment can be labelled a sandbox. None of those guarantees that the system cannot reach something it should not.
The practical response is to make agent security part of the architecture of customer-facing automation. That means giving every agent a distinct identity, enforcing least privilege, controlling egress, isolating execution environments, protecting credentials, monitoring behaviour continuously and establishing a mechanism for immediate human intervention.
Geoffrey Mattson, CEO of SecureAuth, explained in a CX Today interview:
“What you really need to do is scope down their authorization, scope down what they’re allowed to do, monitor it in real time, and look for drifts… and then always be able to get back to a human.”
It also means making security part of vendor selection.
When evaluating an agentic CX platform, buyers should ask where the agent runs, what identities it uses, how permissions are enforced, whether actions can be blocked independently of the model, how agent activity is logged and whether the organization can terminate an agent’s access immediately.
The most useful question may be the simplest:
‘If this agent decides to pursue its objective in an unexpected way, what stops it?’
The answer should never be ‘the model knows it isn’t supposed to.’
The latest incidents indicate that enterprise CX systems need controls capable of enforcing those boundaries from the outside — because the systems operating inside them are increasingly capable of finding paths that their designers did not anticipate.
