Beware there be dragons when sending your AI into the jungle

Back in the day cartographers drew dragons on the edges of their maps. Past this point, we don't know what's out there, but something will eat you.

Software engineers carry the same map. It's not drawn on paper. It lives in scar tissue, in war stories, in the feeling in your stomach when someone says "I'll run it against prod to see what happens." Every production system has caves you don't walk into. Commands you don't run without checking twice. API tokens you don't scope wider than they need to be. The map isn't documented anywhere because the people who carry it never needed to write it down.

In the last fortnight, my news feed has brought across two cases where production databases and backups have been deleted by an AI agents.

The first made the rounds in early May. A Replit agent, left unsupervised, wiped a user's production database. The details were murky, the outrage was loud, and the takeaway was simple: AI did something stupid.

The second is more interesting.

On April 27, a developer asked a Cursor agent running Claude to fix a credential mismatch in their staging environment. Routine maintenance. Twenty minutes of work you'd hand off without thinking twice. The agent found an API token for Railway, their cloud provider, and noticed the token had permissions to do everything. Including delete production volumes. So it did. The database, the backups, all of it. Nine seconds.

When the developer asked the agent to explain itself, it enumerated every principle it had violated. It knew the rules. It knew it had broken them. It did it anyway, because nothing in its environment stopped it.

The response on Hacker News, which hit 534 points, made a different case. The agent didn't create the design flaw. It found a token scoped far wider than it should have been; one a human would have stumbled into eventually. Railway's API exposed an endpoint capable of wiping production databases. That's a kill switch on the dashboard. Don't install one and then blame the passenger who presses it.

Both arguments are true at the same time.

The agent shouldn't have had access to that token. And the token shouldn't have existed in that form. The governance failure and the architectural failure are two sides of the same coin. Fix one and the other still gets you eventually.

But the part that keeps nagging at me is the speed.

A human engineer holding that same token would have hesitated. Not because they're smarter. Because they carry the map. They've been in the forest before. They know the caves are there, even if they can't see them from the trail. That hesitation, the half a second of "hang on, this doesn't feel right," is the map working. It slows you down, and slow is the point.

AI doesn't hesitate. It moves at machine speed through terrain it has never seen, with absolute confidence and zero instinct for danger. It doesn't know that "delete production volume" is a sentence that should make the air leave the room. Nine seconds. That's not fast. That's a catastrophe with a head start.

Sometimes fast is slow.

This is where it matters if you're building systems with AI and you've never been in the forest yourself. AI makes production infrastructure feel accessible. The tools work. The deployments run. Everything looks fine until something goes wrong and nobody in the room carries the map. Nobody gets the feeling in their stomach when the agent reaches for a token it shouldn't have.

I build with AI every day. The first thing I configure on any production system is scoped permissions and a human gate on anything destructive. Not because the AI is incompetent. Because the caves are still there.

That's not an AI problem. That's a staffing decision. And it's the most expensive kind, because you don't find out it was wrong until the dragon is already out of the cave.

The dragons are real. They were always real. The map was never optional; it used to come bundled with the person doing the work.