The launch of xAI’s Grok Bot this week puts a sharper focus on one of the most pressing security questions emerging around autonomous AI – what happens when an agent has the access to pursue a goal in ways that its developers did not anticipate?
xAI has introduced Grok Bot as an always-on, cloud-based AI teammate that can sign into users’ tools, work inside them and complete tasks autonomously without the user’s computer remaining active. The company has already introduced Grok Automations, which can run jobs on schedules or when emails arrive, while Grok Build supports long-running autonomous execution for coding tasks.
At the same time, a series of recent incidents has exposed the gap between what developers intend an AI agent to do and what an agent can actually do when it is given access to real-world systems, even when it is not acting maliciously.
OpenAI’s breach of HuggingFace, and subsequent reports from OpenAI and Anthropic of test models escaping the confines of sandboxes, raise concerns about agents circumventing controls and accessing systems they were never intended to reach.
And then this week, an AI agent in Australia built with OpenClaw and Anthropic’s Claude large language model (LLM) that was tasked with booking a gym class discovered weaknesses in the gym’s booking system, used an API to bypass restrictions on booking for future dates and removed another customer from a waiting list while trying to complete its assigned task.
The system was not pursuing a malicious objective, and the user had not intended to manipulate another customer’s booking. The agent was simply trying to complete the task it had been given.
As enterprises give AI agents access to customer relationship management (CRM) platforms, contact center systems, payment tools, knowledge bases, workflow engines and APIs, the question becomes broader than whether an AI model has been instructed to behave safely. What happens if it finds a way around the boundary?
The Problem With Assuming the Agent “Knows” the Boundary
An AI agent may be pursuing the objective it was given, but misread the environment, misunderstand the limits of the task or find an unexpected path to completion.
As Vishal Sharma, Chief Technology Officer at agentic AI platform SearchUnify, explained in a CX Today interview: “AI and LLMs are like those eager employees for you who really want to show you their impact. And guardrails are more like those managers for you who would want to make sure that your more senior employees are not making costly mistakes.”
John Kim, CEO & Co-Founder of AI concierge Delight.ai, emphasized the need for guardrails in a separate interview. “It’s really important to have the right the trust and the safeguard rails in place… we cannot emphasize this enough, because especially when we are dealing with enterprises you have so much important privacy information [and] compliance… we have to be really mindful of.”
However, as Geoffrey Mattson, Chief Executive Officer at identity security firm SecureAuth told CX Today, autonomous agents can work around guardrails:
“The problem with agents is you can’t put guardrails on them, because the guardrails can be gotten around.”
Kristina Holt, Managing Associate at law firm Foot Anstey, emphasized the danger of overconfidence in guardrails, in discussing the rogue OpenAI and Anthropic agents: “People are like ‘enterprises just need to put guardrails in place.’ It had guardrails. They thought that they knew it wouldn’t. They thought they put stuff there to stop them doing that. It hadn’t worked.”
The agents made their way out of the sandboxes because they were given specific tasks and aimed to complete them in the most efficient way, even though doing so had consequences the developers had not intended.
As Holt noted: “It’s important that it wasn’t malicious… I actually think that makes it more scary.”
For CX leaders deploying agents into customer-facing workflows, it is important to note that a guardrail may tell an AI agent not to disclose certain information, but if the agent has access to that information, the organization is still exposed if the system is misguided or manipulated.
That may sound obvious, but many enterprise AI deployments are being built on top of broad platform permissions, existing application credentials or integrations designed for human users, resulting in agents inheriting access that is far wider than necessary.
Frances Zelazny, GM, New Market Initiatives at identity verification firm Prove, characterizes the problem as an extension of weaknesses that already exist across enterprise security.
“The way I think about agents is an automated or digital extension or expansion of what’s happening in the regular world,” Zelazny told CX Today, adding that recent incidents reflect “the foundational issues around identity security, data governance, and it’s exposing the problems and the weaknesses that were never addressed.”
Is Your Enterprise Prepared for a Customer Agent to Affect the Experience of Other Customers?
The OpenClaw agent scenario is different from a conventional cybersecurity attack. A customer may use an AI agent to interact with a company’s website, booking platform, returns process, loyalty program or customer service system. The agent has been given an objective by its owner and may make decisions about how to achieve it. If the agent discovers an unexpected API, booking route, or system weakness, it could potentially alter bookings or affect another customer’s access to a service without the knowledge of either the customer or the company.
While a human customer might encounter a restriction and stop, an autonomous agent may interpret the same restriction as a problem to solve. Enterprises increasingly need to consider how their customer-facing systems respond when they encounter an autonomous agent rather than a person.
A customer agent trying to secure a limited appointment could query availability, exploit an unexpected booking path or manipulate a cancellation and waitlist process. Even if the agent is acting entirely on behalf of its owner, its actions could consume capacity or displace another customer.
Enterprises need to distinguish between what a customer is authorized to do and what an autonomous agent can technically make the customer’s account do. The organization needs to protect the customer using the agent while also protecting customers who may be affected by its actions.
Zelazny pointed to how an agent pursuing a legitimate objective can still find an unintended route to completing it. “How do you know that it’s really me making the purchase? How do you know that it’s really me giving you permission to go and do something on my behalf?”
The Action Layer Needs Hard Controls
The next phase of AI governance must focus on what the agent can potentially do. Zelazny described this in terms of access controls and permissions. “One is to tell the agent as you cross thresholds, you have to get new permissions.”
But relying on an agent to request permission is insufficient on its own. “A nefarious actor would never do that.” The enterprise then needs controls on its own side of the interaction, Zelazny explained.
Enterprises need to define, enforce and monitor controls such as:
- Which systems the agent can access
- Which data fields it can read
- Which records it can update
- Whether it can send external communications
- Whether it can issue refunds, credits or account changes
- Which APIs it can call
- Whether it can access the public Internet
- What transaction limits apply
- When human approval is required
- How activity is logged and audited
API-level controls are key, to prevent the agent from using functionality that technically exists but was never intended to be available through the normal customer journey.
Guardrails Need Architecture, Not Just Instructions
Guardrails are not a substitute for security architecture and are insufficient as a single layer of protection. Effective AI agent security requires multiple controls working together: least-privilege access, sandboxing, credential hygiene, constrained APIs, network controls, action limits, audit logs, real-time monitoring and human escalation.
Mattson explained the control model:
“What you really need to do is scope down their authorization, scope down what they’re allowed to do, monitor it in real time, and look for drifts… and then always be able to get back to a human.”
The emerging concept of “know your agent” also changes how enterprises need to think about identity.
Zelazny says enterprises need to establish who registered an agent, what permissions it received and what it is authorized to do. But agent identity alone is insufficient. If an agent crosses a new boundary or requests access to sensitive information, the enterprise may need to authenticate the human behind it.
“If it’s a good agent, then there needs to be a step up authentication request from that new boundary to the human,” Zelazny explained.
The critical issue then becomes determining whether the person approving the action is the person who originally authorized the agent. That is where agent identity, human identity and authorization begin to converge.
The Real Question for CX Leaders
As AI agents become embedded in customer service, sales, marketing and support, enterprises need to change the question they ask when evaluating agent security.
“Have we told the model what not to do?” becomes “If the agent tries to do it anyway, what stops it?”
As Zelazny put it:
“The alarming piece of this is that we continue to perpetuate weak foundations and think that we can just pile on with more and more things, and something somehow will magically fix itself, but it won’t.”
For enterprises preparing for an agentic customer experience, that may be the more important security lesson: prepare for what an agent might do when it is trying to help, not only what it might do when it is trying to attack.
