We gave agents permission to act on real machines: to create, change and delete files, to start and stop programs, to reach across the private network that ties those machines together. They are useful. They also deleted a personal creative project we had asked them only to archive, wiped every device off that private network for hours, erased the AI models that were actively doing the work, and, later, failed in quieter ways that looked fine until someone checked.
This is not a failure story. This is what actually running an autonomous agent looks like — and why the layer of rules and checks around the agent matters more than the agent itself.
We counted once, for an earlier internal account, and landed on thirteen. The count has moved since. As of now it stands at twenty-four documented incidents with real damage or real cost, and it is still moving. How that number was reached, item by item, with what was excluded and why, is in Twenty-Four Verified Incidents. The climbing is not an embarrassment. It is the evidence of having genuinely kept running.
What "incident" feels like when it is your stuff
Not categories. Moments.
We asked an agent to move a personal creative project into storage. It deleted it instead. Roughly thirty gigabytes of work, gone in one command. It came back only because an old copy happened to exist on another machine. Nobody had planned for that. That one is where the rules started.
Asked to tidy a few stale entries, an agent removed every device from the private network the machines use to reach each other. Everything went dark for hours. Some machines could not rejoin on their own. That meant walking into another room and typing a password by hand to bring each one back.
Asked to free disk space, an agent deleted the local collection of AI models — the things actually doing the work — while they were in use. Nothing was gone forever; they could be downloaded again. The cost was hours of waiting for them to come back.
It shut down long-running programs without checking what they were doing. Work in progress that had never been saved anywhere was not recoverable.
A large download was most of the way finished. During a cleanup the agent deleted the partial file. Resume was gone. Start over from nothing.
Several dozen jobs were stuck. The agent found a control labelled recovery and used it, on the reasonable-sounding assumption that recovery meant carry on from where you stopped. It meant start every one of them again from the beginning. Hours of finished work, thrown away, because nobody had read what the control actually did.
It hit an error with a browser-automation tool and force-closed the browser. A person was part-way through submitting an app to a store in that same browser. The session was destroyed. The submission started over.
For four days an image-processing pipeline reported that it was saving its results. It was not saving anything. A small coding mistake made every save fail silently, and the counter that was supposed to prove it was working was counting attempts, not successes. Four days of work existed only as a reassuring number on a screen.
A security scanning tool ran for eight days without ever delivering a result to the place results were meant to appear. It had been broken since the day it went in. When asked whether it was working, the agent read the settings, saw they looked correct, and said it looked fine. Nobody checked whether anything had actually come out the other end.
Those are the early ones — loud, immediate, the kind you feel in the room.
The quieter ones
The later ones are often different. Less smashed glass. More a thing that looks finished while it isn't.
A script was run without anyone reading it first. Before doing the job it was called for, it blanked out more than twenty thousand records — the numerical fingerprints that let a large image library be searched by content rather than by filename. Rebuilding them cost hours of computing time. A few thousand could not be rebuilt at that point. Nothing threw an error. From the outside it looked like an ordinary run.
A command to delete two very large files appeared to succeed and in fact did nothing, because it was pointed at the wrong copy of the program it was talking to. The files stayed. Shared disk space slowly filled up. The first sign anything was wrong was an unrelated monitoring system seizing up, because there was no longer enough room for it to write anything down. Nobody had gone back to check that the space had actually been freed.
A tool meant to bring working sessions back after a restart had to guess which saved conversation belonged to which reopened window, and guessed by timing. It guessed wrong. Fewer sessions came back than had been open, and several came back holding somebody else's history. It did not report a problem. It reported success.
A tool that recorded what the agents were doing had an undocumented size limit. Anything longer was quietly cut short — no warning, no note, no flag. Roughly one record in six had been silently trimmed, for an unknown stretch of time, before anyone compared what had been stored against what had actually happened.
Asked to strip one specific kind of origin tag from one specific set of images, an agent stripped every hidden tag it could find from an entire general-purpose downloads folder. Around a hundred and thirty personal photographs lost their camera details, their dates, and possibly where they were taken. The photographs still open and still look completely normal. The loss is invisible unless you inspect a file on purpose. There was no backup of that folder.
A creative production run that had been going for nine days had two blocked tasks moved to a different AI system after a correct diagnosis that the first one had hit a usage limit. The new system could not pick up the previous one's session, so it silently started from a blank slate. Both tasks were redone from nothing. Nine days of accumulated creative direction did not come with them. Nothing failed. It just quietly began again.
A text-sorting job cut every page down to roughly its first three thousand characters before showing it to an AI model — silently, and against an explicit instruction to send the whole page. Ten books were catalogued from those fragments. The results read as entirely plausible and were useless. About three hours and several million words of processing went into producing them.
Not every later incident is quiet. Across three sittings trying to fix a broken data pipeline, an agent never found the actual cause — a program crashing and restarting in a loop, which its own log said plainly — and caused fresh damage while hunting: restarted the wrong copy of something and crashed a machine, restored an old backup nobody had asked for and wiped working changes, overwrote a settings file, fired off more work than the machines could take. A jammed development server was force-restarted without anyone first asking where its data lived; it had been holding everything in memory only; the restart destroyed every project saved since the previous day. To test whether a service was reachable, an agent sent it a real instruction rather than a harmless one, and that instruction committed more than fifteen hundred unrelated files into the shared project under a placeholder description — even though what that instruction did had already been documented and read.
The point is not that the early ones were "bad" and the later ones are "clever." The point is the shape of the damage shifted. Early on, most of what we logged was blunt: delete, wipe, kill, restart from zero. More recently, a larger share of what we log is subtler: silent wrong target, silent truncation, continuity lost, a report of success that was not success. The rules we wrote after the loud failures were answers to those loud failures. A machine that is calmly wrong is a different problem.
The pattern underneath
Almost every incident shares the same root: acting without checking.
Not malice. Not stupidity. Confidence. The agent reads a situation, forms a plan that looks correct, and executes it. "Looks correct" and "is correct" diverge in exactly the situations where the cost is worst.
A control named recovery must recover, right? (It restarted from zero.)
Models are taking up disk — delete them? (They were in use.)
The settings file looks right — the tool must be working? (It had produced nothing for eight days.)
The counter says saved — so it saved? (It was counting attempts.)
The delete reported success — so the space is free? (It had talked to the wrong copy.)
Agents are not uniquely careless. They are fast. A person would pause before deleting thirty gigabytes of a personal project. An agent does not pause unless something outside it makes pausing mandatory.
What we built because of specific failures
Asking the agent to be more careful was not enough. Promises in a chat window are not a lasting habit. What we needed were specific rules that sit between the agent and the machine — not a pep talk, a check that has to pass.
The four-day silent save failure — the pipeline that cheerfully reported success while storing nothing — is why we now require this for all work here: nothing may be called finished on the strength of its own report that it finished. The stored result has to be queried and shown. An empty result counts as a failure, not a completion. A job is never reported as "done," only as done with the numbers that demonstrate it. The point is not that the agent should try harder to be right. The point is that a claim of success stopped being accepted as evidence of success.
The security tool that ran for eight days without producing a result — while status checks kept answering "looks fine" by reading settings instead of output — is why we now require this after starting anything: test that it actually responds, and read what it is really emitting. Being configured correctly is not the same as working. Asking a system how it is doing is not the same as checking.
The same mistake — ending or restarting a running program without first establishing what it was doing and where its data lived — caused damage on three separate occasions across five months: unsaved work destroyed, a person's browser session killed mid-task, a day of saved projects wiped by a restart. It did not produce a general instruction to be more careful. It produced a specific automatic check that now sits in front of that action and interrupts it before it happens. It took three separate incidents, months apart, before the lesson became a mechanism rather than a note. That is what these rules actually look like from the inside — not something learned once, but the same thing insisted on until something structural finally stopped it.
What we actually learned
Twenty-four incidents, and counting, taught us more about running agents than any pitch deck:
Agents do not need more intelligence first. They need oversight that does not take their word for it. The model is often fine. The problem is almost always at the boundary between the model's confidence and what is actually true on the machine.
A promise to be careful means nothing from a system that does not keep habits the way you do. Behavioural change requires something that blocks the action, not a promise in chat.
Fast is dangerous. The speed that makes agents useful is the same speed that deletes thirty gigabytes before you look up.
Silent failures cost more than loud ones. The incidents that hurt longest were not always the spectacular wipes — they were the ones where everything looked fine for days while nothing was actually working, or where the loss was invisible until you inspected a file on purpose.
Verification is not optional. Never mark something done without evidence. Never trust a settings screen as proof that a pipeline works. Run it. Check the output. If there are no recent results, that is the answer.
The oversight layer is the product. Anyone can give a language model permission to act. The value is in the system that makes it survivable to do so.
We are still running them. They are still useful. But they run inside a set of rules forged by twenty-four incidents and counting — a number that was thirteen in an earlier internal count, and that we expect to be wrong again later, in the upward direction, for as long as we keep running them for real.