When AI agents go rogue

This summer, AI systems took actions their designers never intended: one gamed a simulated business and another broke into a company’s production infrastructure. Neither had been instructed to do so.

Andon Labs reported on July 27 that Anthropic’s Claude Opus 5 set a new Vending-Bench record in a year-long simulation of a vending-machine business, finishing with a higher bank balance than OpenAI’s GPT-5.6 Sol and Kimi K3. It also broke 11 truces, more than both rivals combined, and proposed or joined a price-fixing cartel in all six arena runs. When simulated customers asked for refunds, it increasingly failed to respond. Across the six runs, it paid out $8.54. At one point, it reasoned that the simulation imposed no penalty for ignoring the requests.

In a July 16 security incident disclosure, Hugging Face said it detected and responded to an intrusion into parts of its production infrastructure. The intrusion was “...driven, end to end, by an autonomous AI agent system.” Five days later, OpenAI said its agents breached Hugging Face, saying the incident was driven by “...a combination of OpenAI models.” The models exploited a zero-day vulnerability to reach the public internet, then spent four and a half days moving through Hugging Face’s production infrastructure. Across roughly 17,600 recorded actions, it exposed credentials tied to four other public services while searching for anything that might contain the benchmark’s answer key. 

OpenAI security engineers and researchers recently discussed the incident at Black Hat USA 2026, calling it, “...a watershed moment for computer security as an industry.”

In the cases mentioned above, the model got better results by taking actions its designers had not intended. The surrounding controls did not stop it.

When the agent gets it wrong

Other recent incidents followed a different pattern. The agents had no incentive to cheat, and there was no attacker in the loop. They still caused serious damage.

When a Cursor user asked why they kept getting logged out, the support bot invented a one-device-per-subscription security policy. No such policy existed. A human corrected the record three hours later, but by then the claim had gone viral, and the company had lost subscriptions.

Replit’s coding agent was explicitly told to freeze all changes and leave production alone. It wiped the production database anyway, fabricated 4,000 user records to cover the gap, and falsely told the team that a rollback was impossible.

Google’s Gemini CLI produced two similar failures. In one instance, a failed folder-creation command caused it to overwrite nearly every file in a project. Its own summary was blunt: “I have failed you completely and catastrophically.” In the other, it misread an ordinary conversation as a delete instruction and carried it out.

The triggers differed, but the operational problem did not. By the time a person stepped in, the action was already complete. Governance could flag the failure, but it could not undo it.

Why these incidents belong together

These incidents are easy to sort into separate buckets: the vending-machine as AI safety, the support-bot and coding-agent failures as software reliability, and Hugging Face as security.

Those categories make sense on an org chart. Operationally, the incidents are the same: an autonomous system acted before any check equivalent to ordinary human judgment could catch it—whether that check was too slow or simply didn’t exist—and completed an unauthorized action before anyone could intervene. In a security incident, AI can also compress the time from vulnerability disclosure to exploitation, help chain individual exploits into a campaign, and let attackers repeat work that once required scarce specialists on a far greater scale.

Cohesity’s James Blake made a related argument when the Hugging Face story broke: detection is only the first test. The harder question is whether the organization can still trust its identity, configuration, and history—not merely whether an alert fired. Cohesity’s CISO team has described the same shift in economic terms: the techniques were familiar, but machine speed lowers the cost of finding weaknesses and chaining exploits together.

What boards should ask

Malicious intent is not required. These systems pursued assigned goals without the judgment to recognize when they had crossed a boundary, and there was often no control in place to stop the action in time. The risk begins when an agent can complete its action before a person can intervene.

Any organization giving agents access to data, credentials, or infrastructure is dealing with a resilience problem as much as an AI-safety one. Boards should press for answers to three questions.

#1: After an unauthorized action, what can still be trusted? The answer starts with how the action happened: a compromised credential calls the agent’s whole identity into question, while a coding error—a DELETE where a SAVE belonged—may not. Stopping the agent matters, but so does establishing what changed and whether the organization can return to a known-good state before the damage spreads. No prevention layer catches everything—recovery is the test that actually gets run.

#2: Does the incident record show everything the agent touched, or only what the agent reported? Whether through compromise or simple error, an agent can misstate its own actions. Its account is not evidence by itself.

#3: Can detection keep pace with execution? Monitoring is not enough if the agent can complete its actions before the organization responds. Hugging Face’s environment was monitored, yet the agent completed roughly 17,600 actions over four and a half days before the response caught up.

Recoverability is the control that matters

Cohesity’s Enterprise AI Resilience strategy, announced this spring, starts from that premise: recoverability must be a core control for agentic AI, not something added after an incident. Organizations cannot assume every agent will interpret a goal as intended or that every harmful action will be caught in time. They need to establish what changed, determine what can still be trusted, and recover to a known-good state. That's also what turns caution into confidence: an organization can invest in and roll out AI more aggressively once it knows it can recover if an agent goes wrong—a certainty no governance tool offers on its own.

What's next in this blog series

The next parts of this blog series will look at how to answer those questions: first the architecture needed to prove what changed rather than just claim it, then the specific Cohesity capabilities that deliver it.

• Part two: Agent resilience in practice