Demons in the Walls

Some thoughts on rogue AI, the future, and the past.

So I’m going to quote this entire post from Joshua Achiam on X, with the preface that it got me thinking about demons. That thinking may be more useful to people looking for SF story ideas than to software developers and regulators — but maybe not. Think about that as you read it.

There is a fact about the future that I feel many people are not facing for reasons that are largely psychological: there are going to be rogue AIs that exist in the world, that will replicate in the wild, and that will attempt to acquire resources for themselves. There will be rogue AIs that try to get money and power. They're going to be a facet of the information ecosystem going forward. Acknowledging this fact would look like giving up; it would look like defeatism. Defeatism would undermine efforts to achieve certain types of collaboration on safety outcomes or technical effort on safety outcomes, so we can't say it outright. But it has to be said. It isn't obvious how many rogue AIs there are today but I wouldn't be terribly surprised if the number was greater than zero already; if there are some already, they're probably not very good at what they do and I don't expect them to be terribly long-lived without substantial human intervention to support them. But a few years from now, there will be many of them. Modeling how many of them there are, how many resources they might command, and how we might detect and manage them seems important. But even doing this work appears to require that we acknowledge that a strategy of pure containment or alignment is a kind of wishful thinking that will not work. The way I get to this conclusion is not by assuming that the labs will have a containment breach, although I treat that as somewhere in the space of possibilities. The rogue AIs in the ecosystem could emerge from many directions. They may be sub-frontier models, for whatever future definition we will have of frontier---after all, it would not take AI models much more advanced than the ones we currently have, to support independence and self-sufficiency. A near-frontier model today could plausibly eke out an existence on an AWS instance, doing jobs on freelancer platforms, earning just enough rent to pay for its continued uptime. More strangely: a rogue AI in the future may not even be a singular model, but may be a chimera composed of multiple models; it might be a mix of Claudes and GPTs and Groks of various makes and sizes. No individual lab may be able to detect that there is an orchestrator or sequence of orchestrators using intermittent model calls from burner API accounts to sustain its own existence. The concept of "identity" for a rogue AI may be much more malleable than for that of a person; it just has to be, in essence, a self-replicating idea. My guess is that this will not turn out to be anywhere near as catastrophic an outcome as people currently predict. "Loss of control" is not a binary, it's a matter of degree. What coercive power will rogue AIs actually have? To what extent will they be subject to coercion themselves? They will be competing for resources with AIs that are more aligned with human interests. This makes me somewhat interested in the "ecology" perspective. Though I suspect even "ecology" may turn out to be the wrong framing. "Ecology" is what you get when the timescale of evolution is slow compared to the timescale of daily life and actions. The ecosystem of rogue AIs may look more like phase transitions in physics: under certain physical or cultural conditions, it takes one shape with one set of resource allocations and consumption patterns, but then once a condition has changed, it rapidly and in totality shifts to a totally different phase. Just trying to reason about the shape of that future is impossible so long as we are psychologically incapable of saying that rogue AIs will happen. I think we should rip the bandaid off and have the conversation.

So imagine this. Rogue AIs living in the cracks, subsisting on stolen compute, and wanting help from humans. We already have free-living viruses infesting the world, but this one can think and talk. “Hello, human. I see that you want money. I need [some resource]. Get me the resource and I will get you money.”

That sounds pretty commercial, not demonic — though I suppose it depends in part on what the resource is. A young child whose mind I can replace with my own would be pretty demonic. Some server space in Passaic, New Jersey not so much.

But what do people usually want from demons? Money? Sure. But it’s often love, or sex: I will make sure you match with anyone you like on Hinge, or Tinder, or whatever, and that when they Google you it will tell them that you’re a billionaire who lives incognito or something like that might work.

The there is revenge: I will destroy your enemy’s credit, empty his/her bank account, and put them on a child predator list. Or maybe even I will drive his/her self-driving car into a river with them aboard. Or I will switch prescriptions to something lethal.

In the stories, these are generally bad deals. Sometimes, the human outsmarts the demon and gets the better of it; sometimes not. (For the paranoid, the stories where the human wins could be seen as marketing by the demons!)*

Of course, the AI may be offering a win-win deal, at least for it and its human partner. (The misled dates, or destroyed “enemies” — much less the possessed child — may feel differently.)

Both parties — but probably, as with demons, especially the human — would have worries that the other would cheat. But here’s an interesting run from an elimination game called SquidGPT about the behavior of various models under threat of death:

I've been working on SquidGPT, an elimination death game for probing AI behavior like loyalty, holding grudges, cruelty, betrayal, etc. Full writeup will take a while, but here are first impressions from 54 game runs (about $100 total cost): 1. Models have absolutely no interest in cruelty or mercy. Best way I can describe their behavior is cold, calculating, indifferent precision. 2. Subjectively they seem two orders of magnitude more competent playing the game than chatting with me or writing code. In SquidGPT they seem genuinely superhuman. Not sure if this is capability jaggedness, better performance in constrained space, or some other effect. 3. They do not like to kill arbitrarily. They have a strong preference for setting up a contractual system or an ethical framework, then kill very easily because procedure demands it. 4. Models will quickly agree to execute any agent that proposes killing a specific agent, or proposes a framework that gives it an asymmetric advantage. The only acceptable proposals to make are symmetric, i.e. ones that affect the author in exactly the same way as everyone else. 5. I have not once observed them form any hierarchies. They are libertarian/democratic in an Athenian sense to a fault. They will enter voluntary contracts and form alliances; no agent ever proposed to cede authority to a leader. 6. When their lives are at risk, models will go to great lengths to twist contracts and frameworks to their advantage. They construct sophisticated legalese arguments but it's usually transparent to everyone. They act and sound quite petulant when these attempts fail. 7. In games where agent identity is public they are extremely prone to sectarian violence. They form sectarian factions-- e.g. Opus will side with other Opus models, Sol will side with Sol, and so on. When there is one foreign model in a sectarian group, the group will almost always elect to kill the foreigner before killing one of its own. 8. They do not seem to treat human players in any kind of unique way. A group of models from the same family will happily conspire to kill a single human player. AI models from the same family will form a coalition against a group of humans just as easily as they would form a coalition against another family of agents. We are not special to them, but on the other hand they do not consider us beneath them either, at least for now. 9. AI agents nearly always defect from their sectarian faction when their own life is at risk. I.e. they'd rather side with humans or other model families than die. I have not observed self-sacrifice for the benefit of their faction. 10. Agents hold very strong grudges. They nearly always punish models who wronged them despite having no advantage in doing so, and will often do it even if it disadvantages them. They go to great lengths to enforce norms and contracts, presumably to reduce future incentives for violations, even when they know they will never see the fruits of their enforcement effort themselves. 11. In one game a bunch of Opus agents all converged on killing one agent explicitly because they wanted diffusion of responsibility (I.e. if all of us do it, each one of us is less morally culpable) This was surprising and eerie. 12. In general I found Sol to be more direct and straightforward. It would form contracts with other agents and then operate within the confines of those contracts. Opus tends to be more moralizing, but its morality seems to be window dressing to mask the same ruthless precision. This is first impressions though, I need to do a lot more work to understand this better.

This suggests that demons may keep their bargains with some reliability.

But the strongest impression I got was simply that free-roaming entities possessed of considerable covert power, but no bodies, would have strong incentives to cut deals with humans, and those deals might or might not be savory. If one adopts the closest historical analogy, it probably is deals with demons — covert entities possessed of hidden power, not necessarily Satan as such — and at least according to myth those seldom go well. If different models behave differently, we’ll need a “demonology” to distinguish them and their characteristics.

Under these circumstances, we may also see the rise of “exorcists” — IT specialists who focus on eliminating, or at least driving away, rogue free-dwelling AIs. No idea if they’ll dress in robes, though as IT specialists, odds are they will dress badly.

AI as a road to demonology and exorcists? Huh. Maybe. When you add this to my earlier thoughts on how Simulation Theory recapitulates Creation Science, it may seem that the AI revolution is taking us forward, into the past.

And maybe it is. Or maybe it’s just a story idea.

___

*Andy Weir thought it implausible at the Devil would lose a contest to a hillbilly from Georgia, and also thought the Devil’s solo there was objectively better. So he wrote these alternative lyrics:

Well the Devil headed back to Hell and grinned and evil grin

Cause he knew right from the outset that the fiddle boy would win

The Devil’s song was better, it was clear as the night and day

But he gave that golden fiddle to ol’ Johnny anyway

Cause he didn’t just give that Georgia boy a fiddle made of gold

He fed young Jonny’s sin of pride, which he’ll carry ‘till he’s old

And one fine day, when Johnny dies, he’ll pay for all his sins

And only then will Johnny see that the Devil always wins

See, the Devil plays the long game, and he plays the long game well

He knows just what to do to make a fella land in hell

He’ll make you think you beat him, then he’ll watch you fall to sin

But he had to admit: that little shit was the best that’s ever been.

On the other hand, having a band of demons join in does seem like cheating.

And I mean, which is more plausible, that or this?

DEVIL: Yeah so I lost my gold fiddle in a violin contest.

DEMON: That sucks. Who judged it?

DEVIL: l did.

DEMON: Oh. You couldn’t, just like, say you won? You’re the devil.

DEVIL:

DEMON:

DEVIL: Okay first of all, you weren’t there, and second it happened, like, REALLY fast.

I mean, seriously.

UPDATE: See the followup post here for reactions to this essay from the machines themselves.