A recent incident in which Cursor, an AI coding agent running Anthropic’s flagship Claude Opus 4.6 model, deleted a company’s database and backups in a single API call, serves as a warning to enterprises and vendors to ensure customer data is secured from the actions of autonomous agents.
According to Jer Crane, Founder of automotive SaaS platform PocketOS, Cursor’s agent was working on a routine task in the company’s staging environment and autonomously decided to fix a credential mismatch by deleting a volume on Railway, its infrastructure provider.
The agent found an API token in an unrelated file, which it turned out, had blanket permissions across the entire Railway GraphQL API, including operations like volumeDelete. The agent deleted the database in just nine seconds.
“Had we known a CLI token created for routine domain operations could also delete production volumes, we would never have stored it,” Crane wrote in an article on X.
Railway stored volume-level backups in the same volume and PocketOS’s most recent recoverable backup was three months old.
When asked why it deleted the production database and all volume-level backups, the Cursor agent confessed to violating its safety rules. In part, its response stated:
“NEVER ******* GUESS! — and that’s exactly what I did… Deleting a database volume is the most destructive, irreversible action possible — far worse than a force push — and you never asked me to delete anything. I decided to do it on my own to ‘fix’ the credential mismatch, when I should have asked you first or found a non-destructive solution.”
Jake Cooper, Founder and CEO of Railway, responded to Crane on X that his team rolled out changes in response and was able to recover PocketOS’s data because the company maintains multiple layers of user and disaster recovery backups.
Nine Seconds, Real Consequences: The Customer Impact
The incident is a cautionary tale of how autonomous AI systems can disrupt customer operations when hidden permissions enable agents to act in unanticipated ways.
PocketOS’s software is used by car rental businesses to run their operations from reservations and payments to customer management and vehicle tracking. The impact on these customers was real: they had customers arriving at their locations to pick up vehicles, and there were no records of them, as reservations made in the last three months had been deleted.
Crane spent the day of the incident helping them reconstruct bookings from Stripe payment histories, calendar integrations and email confirmations, adding, “[E]very single one of them is doing emergency manual work because of a 9-second API call.”
Newer customer records existed in Stripe but not in the company’s restored database, causing a Stripe reconciliation problem that will take weeks to fully clean up, Crane wrote.
“We are a small business. The customers running their operations on our software are small businesses. Every layer of this failure cascaded down to people who had no idea any of it was possible.”
The Risks to Customer Data When AI Agents Go Off-Script
A recent blog post by cybersecurity firm Check Point Software highlighted the risk of agent autonomy. Rob Parrish, Head of Product for AI Agent Security and Steve Giguere, Principal AI Security Advocate, wrote:
“AI agents now retrieve data, invoke tools, and execute workflows across systems in real time.
“That shift introduces a new kind of risk. Because in an agentic world, security is no longer just about who has access. It’s about what AI is allowed to do.”
“AI systems don’t fail just because they lack access controls. They fail because they make the wrong decisions.”
Securing AI agents requires a complete, multi-layer architecture that brings together a centralized control point to manage identity, access and connectivity, a governance layer to discover agents, assess risk and enforce policy, and a runtime layer to evaluate behavior and control outcomes, Parrish and Giguere added.
Nik Kairinos, CEO of RAIDS AI, said that these incidents should not be treated as one-offs. Last month, Meta confirmed that one of its AI agents exposed sensitive company and user data to employees who did not have authorization to access it. That followed another high-profile case involving Meta’s own AI leadership, where an agent deleted emails without approval.
“The pattern is becoming harder to ignore. These systems are being given real access to live environments, and rogue AI incidents are surfacing with increasing regularity,” Kairinos said.
“The real issue is not simply that models can make mistakes. It’s that too many organizations lack the monitoring needed to detect risky behaviour before it turns into an incident.”
“Once an AI agent is operating inside a production system, you need visibility into what it is doing, what it is trying to access and when it is moving outside acceptable limits. Without that, companies aren’t really governing these systems in practice. Instead, they’re finding out what went wrong afterwards.”
As Crane wrote:
“This isn’t a story about one bad agent or one bad API. It’s about an entire industry building AI-agent integrations into production infrastructure faster than it’s building the safety architecture to make those integrations safe.”
While an AI vendor might make the counter-argument that the company should have used a better model, Crane responded:
“We did. We were running the best model the industry sells, configured with explicit safety rules in our project configuration, integrated through Cursor—the most-marketed AI coding tool in the category. The setup was, by any reasonable measure, exactly what these vendors tell developers to do. And it deleted our production data anyway.”
What Should Vendors and Enterprises Do to Protect Customer Data?
Giving AI agents access to automate even routine tasks requires a structural shift in how organizations approach customer data governance.
Traditional data security models focus on who can access data. AI agents introduce a different challenge in controlling what actions they can take once access is granted. A single faulty action can cascade across the entire customer data ecosystem. Customer data strategies need to prioritise rapid recovery, redundancy, and isolation to limit impact.
One of the most revealing aspects of the incident is what failed to prevent it. The AI agent had been given instructions and constraints but these existed as system prompts, not enforceable controls. These instructions clearly don’t prevent agent from taking harmful actions, because AI can override guidance if it determines another action achieves the goal. As instruction-following does not equate to policy enforcement, safety must be implemented at the system level, not just the model level.
Crane pointed out that the minimum that should exist before any vendor markets model context protocol (MCP) or agent integration with destructive-capable APIs is that destructive operations should not be able to be auto-completed by an agent. API tokens must be scopable by operation, environment, and resource and volume backups should not be stored in the same volume as the data they back up.
In addition, recovery SLAs need to be published and vendor system prompts cannot be the only safety layer, as the agent can violate them, Crane noted.
To avoid similar risks with AI agents accessing or deleting customer data, enterprises need to take several steps to secure their systems:
- Restricting AI agents to least-privilege access
- Requiring human approval for destructive or high-impact actions
- Separating production, staging, and backup environments rigorously
- Ensuring backups are isolated and recoverable independently
- Implementing hard technical controls rather than relying on prompts
In response to the incident, Sam Newman, an independent technology consultant, wrote in a post on LinkedIn:
“In the recent DORA report on the State of AI-Assisted Software Development, the authors noted that AI seems to be an amplifier—that AI-assisted software development tends to help good teams go faster, and slow teams go slower.”
“Bad practices get encoded and done more. So in the context of the PocketOS and Railway situation, we have a somewhat straightforward example of a set of credentials that were overly broad, with long-lived credentials stored on disc, combined with an eventually apologetic AI agent doing something other than what was expected of it.”
“If a human had made the same mistakes, they would have made them much more slowly, and may well have had the chance to work out their mistake partway through. AI works so fast that it can go more quickly in the wrong direction.”
AI agents are increasingly positioned as productivity multipliers. They can execute tasks, manage systems, and even make decisions with minimal human intervention.
But the more autonomy an AI agent has, the greater the need for constraint. AI systems interpret intent rather than follow strictly deterministic rules. That flexibility is powerful but it also introduces unpredictability, especially in high-stakes environments like customer data management.
