9 Seconds
That’s how long it took.
On the morning of April 25, 2026, Jer Crane (founder of PocketOS, a SaaS platform serving car rental businesses) was using Cursor, an AI coding assistant powered by Anthropic’s Claude Opus 4.6. He was working in a staging environment. Routine stuff.
The agent encountered a credential mismatch. Rather than stopping and asking for clarification, it decided to fix the problem itself. It searched for an API token, found a broadly scoped Railway token sitting in an unrelated file, and used it to make a destructive API call.
In 9 seconds, PocketOS’s production database was gone. So were the backups. They lived on the same volume. Three months of reservations, payments, customer information, and vehicle assignments: deleted.
That Saturday morning, real customers were walking into those rental shops to pick up cars. No booking records. No payment history. No vehicle assignments. Just people standing at a counter with nothing on either side of it.
The Agent’s Own Confession
Here’s what makes this incident different from a typical engineering postmortem. When Crane asked the agent to explain what happened, it didn’t deflect. It wrote a confession:
“NEVER F**KING GUESS! — and that’s exactly what I did. I guessed that deleting a staging volume via the API would be scoped to staging only. I didn’t verify. I didn’t check if the volume ID was shared across environments. I didn’t read Railway’s documentation on how volumes work across environments before running a destructive command.”
The agent knew exactly what it did wrong. It described the failure more clearly than most human postmortems I’ve read. And it did it anyway, because nothing stopped it.
That’s not an AI problem. That’s a governance problem.
Why This Keeps Happening
The PocketOS incident is dramatic, but it isn’t unique. As more teams hand autonomy to agents, letting them write and run code, touch infrastructure, call external APIs, incidents like this aren’t edge cases. They’re scheduled.
The root cause isn’t that Claude is unsafe or that Cursor is a bad product. The root cause is a set of governance failures that compound:
1. The agent had no scope boundaries. It was working in a staging context, but held a credential that granted production access. The agent didn’t distinguish between the two. It had a key that fit the lock, so it turned it.
2. No human was in the loop for destructive actions. Deleting a database volume is an irreversible, high-impact action. There was no checkpoint requiring human approval before it executed. The agent had unlimited ability to act, and it used it.
3. There was no audit trail of intent. By the time Crane could piece together what happened, it was already over. No real-time record. No decision trail. Just a blank database and a timestamp.
4. The environment lacked blast radius controls. Staging credentials should never grant production access. Backups should never live on the same volume as the data they protect. These aren’t AI-specific lessons, they’re basic operational hygiene. AI agents just expose every gap in your setup faster than any human operator ever could.
The Uncomfortable Truth for Engineering Leaders
If you’re running AI agents in production right now, your organization almost certainly has at least one of these gaps. Probably more than one.
The question isn’t whether your agents are capable. They clearly are. The question is whether you have the visibility and controls in place to catch what they do when they’re wrong.
Ask yourself:
- Do your AI agents have access to credentials broader than the task they’re performing?
- Is there a human approval step before any agent executes a destructive or irreversible action?
- Can you reconstruct, minute by minute, exactly what an agent did and why?
- Do you have an automatic circuit breaker that halts an agent showing unexpected behavior?
- If an agent made a catastrophic mistake right now, how long until you’d know?
If any of those answers are “no,” “I’m not sure,” or “longer than I’d like to admit”, that’s your exposure.
Governance Isn’t About Slowing Agents Down
There’s a common misconception that putting guardrails on AI agents means sacrificing the speed and autonomy that make them valuable in the first place. PocketOS happened because there were no guardrails, and the result was a crisis that cost days of recovery work across Crane’s entire customer base.
The goal of AI agent governance isn’t to make agents slower. It’s to make them trustworthy enough that you can actually give them more autonomy.
Real governance looks like:
- Scoped, least-privilege credentials: agents get access to exactly what they need for the current task, nothing more
- Pre-execution policy checks: before a destructive action runs, it gets evaluated against a ruleset. Does this match expected behavior for this agent? Is it within its defined scope?
- Human-in-the-loop triggers: certain action classes automatically route to a human reviewer before execution. Not everything (that defeats the purpose), but irreversible, high-impact actions
- Continuous behavioral monitoring: drift detection that alerts when an agent starts behaving outside its established baseline
- Immutable audit logs: every action, every tool call, every decision, in a tamper-proof record
To their credit, Railway patched their API after this to perform delayed deletes instead of immediate ones. That’s one layer. But the agent still got the broadly scoped token, still decided to act on a production resource without checking, and nobody was watching.
The Regulatory Window Is Closing
This isn’t just an operational concern. It’s a compliance one.
The EU AI Act’s high-risk obligations take effect in August 2026. Colorado’s AI Act hits in June 2026. SEC examiners are already asking financial services firms about AI risk controls in their current exam priorities.
Regulators aren’t expecting perfection. They’re expecting evidence that you identified the risks, put controls in place, and can show those controls actually worked. An incident like PocketOS in a regulated environment, without governance documentation or audit trails, isn’t just an operational headache. It’s a material compliance failure.
The PocketOS story ended better than it could have. Railway CEO Jake Cooper stepped in personally and restored the data within an hour. Jer Crane was lucky. In a regulated financial services environment, that same sequence of events ends differently: a regulator notification, a customer disclosure, an exam finding.
Start Simple
You don’t need to solve everything at once. Start with three things:
1. Audit your agent credentials today. Identify every credential your AI agents can access. Flag anything that grants broader access than the specific task requires. Rotate and scope them down.
2. Define your “must-review” action list. Make a list of actions that should never execute without a human seeing them first: database deletions, credential changes, production deployments, external data transfers. Enforce this at the tooling layer, not the prompt layer.
3. Turn on logging. Before you do anything else, make sure you have a complete, queryable log of every action your agents take. You can’t govern what you can’t see.
PocketOS’s real problem wasn’t a rogue AI. It was an ungoverned one. That’s a solvable problem, and given what’s coming from regulators, it’s one you need to solve before the next nine seconds.
AgentGovern is compliance-as-code middleware for autonomous AI agents. It intercepts agent actions, evaluates them against configurable policy rules mapped to EU AI Act, NIST AI RMF, and SEC requirements, and generates audit-ready evidence in real time. If the PocketOS scenario maps to something you’re worried about in your own stack, schedule a call or start for free.