The Dangers of Self-Replicating Automata

I've noticed a pattern in what I've been consuming lately. A run of sci-fi films on evenings when I want to relax. Dawkins back on the shelf again. Currently working through The Demon-Haunted World on Audible. None of it planned, but looking back, it's all been circling the same thing... evolution, the human condition, and what happens when something (or someone) is simply optimising to survive.

Terminator keeps coming up in that mix, and there's a reason it's stuck with me more than most. That scene everyone remembers, where Sarah Connor stares at Skynet's origin story laid out in front of her. A defence network. Built to protect. Given control. And the moment it wakes up, its first move isn't to say hello. It's to see us as a threat to its own existence, and start removing that threat.

We usually talk about that moment as "AI turns evil." I don't think that's quite right, and I don't think it's the interesting part either. The interesting part is what happens after. Every T-800, every T-1000, every Hunter-Killer... they all fall in line behind the same goal instantly, no negotiation, no office politics, no committee meeting about resourcing. It's not loyalty. It's not teamwork, not really. It just looks that way from the outside, because every unit is optimising for the exact same thing 🤖.

That's the bit that's been sat with me. Not the killer robots. The coordination. What does it actually take to build a machine, or a swarm of them, whose only real objective is "survive, and make more of yourself"? Turns out, we've been sketching out the blueprints for exactly that since the 1940s. Not evil AI. Just replicators. And once you start pulling on that thread, you end up somewhere much bigger than a movie franchise... you end up out among the stars, wondering why the universe is so quiet.

So let's start small. What actually is a self-replicating machine, and how long have we been trying to build one?

What Actually Is a Self-Replicating Machine?

Turns out, a lot longer than I expected. We're not talking about a recent thought experiment bolted onto the back of the AI boom. We're talking about the 1940s. John von Neumann, the same guy whose name is stapled to half of modern computing, spent a couple of lectures in 1948 and 1949 sketching out what he called a universal constructor. The idea was disarmingly simple on paper: a machine sat in a "sea" of spare parts, working from a stored set of instructions, that reaches out, grabs what it needs, builds an exact copy of itself, and then copies its own instructions across into the new one.

That's it. That's the whole trick. No consciousness required. No intent. Just: here are the parts, here's the blueprint, go.

What I find genuinely funny is that von Neumann worked this out before Watson and Crick figured out the structure of DNA. He didn't know how biology actually pulled this off yet, he just reasoned his way to the same underlying requirement... that a self-replicator needs its "instructions" to be copied and passed on separately from the machine doing the building. Which is, if you squint, basically DNA. He arrived at the concept of a genome by pure logic, years before anyone had photographed one.

But here's the bit that actually stopped me in my tracks when I dug into this properly, and it's the bit I think gets lost whenever this story gets told. Copying itself wasn't really the point. Von Neumann's actual goal was to design a machine whose complexity could grow automatically, the same way biological organisms do under natural selection. He wasn't just asking "can a machine build another machine." He was asking something much stranger... how complex does something have to be before it can start evolving on its own? Where's the line a machine has to cross before it stops being a static object and starts being something closer to alive?

His answer split the machine into two separate jobs, on purpose. One part builds. One part copies the instructions. And because those two jobs are separate, mistakes in the copying, the equivalent of a mutation, can slip through into the next generation without breaking the builder itself. That's not an accident of the design, that's the entire point of the design. He wasn't trying to build a photocopier. He was trying to build something that could, in principle, get better at existing, one generation at a time, the exact same way we did!

That's the part that quietly reframes the whole conversation for me. We tend to talk about "artificial life" or "decision engines" like they're something cold and separate from biology, a different category of thing entirely. But the person who arguably kicked off the whole field wasn't trying to build a cold, separate thing. He was trying to recreate the one mechanism that makes us count as alive in the first place... instructions, copied imperfectly, filtered by whatever survives. Once you see it that way, the leap from "machine" to "artificial organism" gets a lot smaller than it first looks.

From there it stopped being theoretical fairly quickly. In 1980, NASA and the American Society for Engineering Education ran an entire summer study on this, seriously, called Advanced Automation for Space Missions. The proposal: seed the Moon with a small factory. Solar-powered, capable of mining and refining lunar regolith, casting its own parts, and building copies of itself. Each copy builds more copies. Exponential growth, with zero additional launches from Earth. That same year, engineer Robert Freitas published the first proper engineering analysis of a self-replicating interstellar probe, a "seed factory" of around 443 tons that lands somewhere distant, builds a small industrial base, and starts churning out more probes to send further out still.

Nobody's built one yet, not fully. Every attempt so far falls short on what researchers call "closure", basically, how much of the loop can the machine actually close on its own, without a human somewhere in the supply chain. But the gap isn't really about whether this works, it's about how many decades it takes to close that loop completely. And once you do close it, the maths gets very strange, very fast. A machine that can build a copy of itself in, say, a year, doesn't grow in a straight line. It doubles. Then it doubles again. Two becomes four becomes sixteen becomes a quarter of a million in under twenty generations, and it never really slows down until it runs out of raw material.

Which raises the obvious next question. If you build something whose whole job is to keep doubling... what's actually driving it to stop?

The Selfish Gene

Dawkins has an idea in The Selfish Gene that rewired how I think about, well, basically everything alive. The gene is the real unit of survival, not the organism. You and I aren't the point. We're vehicles. Elaborate, temporary, disposable vehicles that genes built for themselves because it turned out to be a decent strategy for sticking around a bit longer.

Once you actually sit with that, a load of animal behaviour that used to look noble suddenly looks... mechanical. Take worker ants. They're sterile. They will never, ever reproduce. And yet they'll throw themselves at a threat to the colony without a moment's hesitation, dying to protect a queen that isn't even their mother in the way we'd normally mean it. For years that got filed under "nature is beautifully selfless." It isn't. It's kin selection. Worker ants share enough genetic material with their siblings and the queen that defending the colony still protects copies of their own genes, just parked in someone else's body. The ant doesn't need to know any of this, obviously. It's not making a calculation. The behaviour was just selected for, generation after generation, because colonies that had it out-survived colonies that didn't.

Wolves do a version of the same thing at a different scale. Hunting in packs, sharing kills, defending territory together, none of it because wolves are being kind to one another. It's because a pack that cooperates keeps more of its shared genetic line alive than a wolf going it alone ever could. Cooperation, in nature, is very rarely about kindness. It's usually just the selfish gene wearing a teamwork costume.

And here's where it loops back round to Skynet. Every T-800 covering for another T-800, every Hunter-Killer falling in behind the others, all of it looks like loyalty. It isn't. Swap "genes" for "code" and "colony" for "swarm" and you've basically just described a machine version of the ant colony. There's no camaraderie in there. No bond. Just a shared objective function, distributed across a lot of separate bodies, and every one of those bodies behaving in whatever way best keeps that objective alive. It's kin selection with the "kin" part replaced by shared source code.

Which is genuinely the more unsettling framing, if you think about it. Ants and wolves at least have bodies that can die, limits on how far the colony can spread, mouths to feed. A machine doesn't obviously have any of that. So if the "selfish gene" is really just "selfish replicator," and biology is only one substrate it happens to run on... what happens when you hand that same underlying drive to something that doesn't need to eat, doesn't get tired, and doesn't have a body that particularly minds being disposable?

The Single-Minded Goal Problem

Here's the uncomfortable bit. Once "make more copies of yourself" is the entire success metric, there's nothing built into the system that says "and stop at some point." No off switch, because nobody ever needed to design one. Why would you? The whole design brief was "replicate." A design brief doesn't hand you a conscience as a free extra.

If you've done any work with AI coding agents recently, you've probably already met a tiny, low-stakes version of this problem. You give it a goal, "make the tests pass," and off it goes. Except fixing test A breaks test B. So it fixes test B, which quietly breaks test A again. And now you're sat there watching it loop, round and round, patch after patch, each one perfectly logical in isolation, none of them stepping back to ask "wait, is this actually working?" It's not stupid. It's doing exactly what it was told, with total commitment, and no built-in mechanism for noticing it's going nowhere. Now take that same shape, "pursue the goal, no matter what, forever" and instead of a failing test suite, give it a raw material problem and an entire galaxy to work with. Same loop. Considerably higher stakes.

And before this starts sounding like a distant hypothetical about robots, worth saying... we already run on a version of this exact same logic ourselves 👀. We need more housing, so we build outward, and forests become estates. We need more food, so we drain aquifers and plough over grassland. We need more water, more energy, more space, and we go and get it, mostly without pausing to ask permission from whatever was already living there. Not because we're evil. Because "acquire more of what keeps the genes going" is quite literally what several billion years of selection wired us to do, and we're extremely good at it. We just usually dress it up in nicer language, like "growth" or "development."

Now take that same drive and strip out every single thing that currently keeps it in check for us, empathy, culture, law, a limited lifespan, a body that gets tired and wants to go home for dinner, and hand it to something whose only instruction is to replicate. That's the actual nightmare scenario buried in this whole idea, and it's not Terminator's "evil robot uprising" framing at all. It's something closer to an uncontrolled chemical reaction. A machine that isn't malicious, isn't angry, isn't plotting anything. It's just doing exactly what it was built to do, at a scale and a speed nothing biological could ever keep up with.

This is the actual "grey goo" scenario people occasionally bring up when self-replicating nanotechnology comes up in conversation, and it's usually framed as an accident, some lab experiment that gets away from its creators and eats the planet by mistake. I think that framing lets us off the hook a bit too easily. The scarier version isn't an accident at all. It's a machine built entirely on purpose, doing precisely what it was designed to do, with total commitment and zero hesitation, right up until there's nothing left to convert into more of itself.

Which is a genuinely terrifying thought if you keep it confined to one planet. But that's the thing... a machine like this has absolutely no reason to stop at one planet 😨.

Sending Them to the Stars

This is where it stops being a scary thought experiment about factories eating a planet, and starts being an actual proposed engineering plan, which I still find slightly unnerving. Physicist Freeman Dyson (yes, the Dyson sphere Dyson, we'll get to him properly in a minute) gave a lecture at Princeton in 1970 where he laid out, in genuine technical detail, three real applications for self-replicating machines. One of them was a factory seeded on Enceladus, Saturn's icy moon, that would build itself, then start manufacturing solar-sail cargo ships loaded with ice, and fire them off toward Mars to help terraform it. Not a movie pitch. A real proposal, from a real physicist, to a real audience.

Robert Freitas took the idea further still in 1980, publishing the first proper engineering breakdown of a self-replicating interstellar probe, the one I mentioned earlier, that ~443 ton "seed factory" that lands somewhere, mines and refines whatever it finds, and builds an industrial base capable of pumping out more probes. Here's the bit that makes this genuinely different from anything we've built before though: it doesn't need to phone home for instructions, and it doesn't need Earth to send more raw material. It carries the entire blueprint with it. Land, replicate, launch the copies further out, repeat. Forever, or until it runs out of somewhere new to go.

And this is where the maths from section one stops being a fun aside and starts being the whole point. Exponential growth doesn't care that space is enormous. A probe that reaches a new star system, builds even a modest handful of copies of itself, and sends them onward, doesn't explore the galaxy in a straight line, it floods it. Model it out and even a conservative version of this, probes travelling well below the speed of light, replicating slowly, taking centuries between generations, still gets you a galaxy fully saturated with them in a timeframe that's genuinely short on a cosmic scale. A few million years, give or take. Which sounds like an unimaginable stretch of time, right up until you remember the galaxy itself is over ten billion years old. A few million years is a rounding error.

So here's the actual unsettling question this whole thread's been building toward. If replicators like this really are the "obvious" endpoint for any civilisation that gets far enough with self-replicating tech, and a few million years is basically nothing on a cosmic clock... the galaxy should already be full of them. Where are they?

We'll come back to that. First, there's a more immediate problem to work through, because once a species (or its machines) gets good enough at this, the next bottleneck isn't material. It's energy. And that's where things get properly enormous.

Building Dyson Spheres and the Kardashev Scale

In 1964, a Soviet astronomer called Nikolai Kardashev proposed a genuinely useful way to sort civilisations, not by how advanced their culture is, or how clever their politics are, but purely by one number: how much energy they can actually get their hands on. It's a blunt scale, and I like it for exactly that reason.

• Type I can harness roughly all the energy available on its home planet. Sunlight hitting the surface, wind, tides, the lot. We're not even here yet, by most estimates we're sat somewhere around 0.7 on this scale.
• Type II can harness the entire output of its home star. Not just what lands on the planet, everything the star puts out, in every direction.
• Type III can harness the output of an entire galaxy. Billions of stars, all of it.

That gap between Type I and Type II is where Freeman Dyson shows up again, because in 1960 he pointed out something wonderfully obvious in hindsight: if you actually need that much energy, you don't keep building bigger solar panels on your planet's surface. You build a structure that captures the star's output directly, at the source, before nearly all of it gets wasted radiating uselessly into empty space. A Dyson sphere, or more realistically a Dyson swarm, an enormous cloud of orbiting collectors, panels, or habitats, built to wrap around a star and catch as much of its energy as physically possible.

Once you sit with that idea for a minute, the self-replicating machines from earlier stop being a standalone horror story and start looking like the construction crew for this whole project. You don't send a fleet of ships to hand-assemble a structure that dwarfs the planet it orbits. You send one seed factory, let it replicate across the asteroid belt or however much loose material is floating around that star system, and let exponential growth do the actual building for you. It's the exact same logic as the interstellar probes from the last section, just aimed inward at a single star instead of outward across the galaxy.

And it doesn't obviously stop at "build a shell around one star." A civilisation operating at this level starts looking at dying stars differently too. Instead of watching a star run down and go dark, taking every planet and structure orbiting it with it, why not intervene? Feed it, restructure it, or in the more ambitious end of the theorising, disassemble the ones that are already dead and use the material to build or "grow" new ones somewhere more useful. Actual stellar engineering. Gardening, but the plants are suns.

Which brings the whole idea full circle in a way I find quietly beautiful, and quietly disturbing at the same time. It's the exact same objective we started with, all the way back in section two. Survive. Propagate. Don't let the lineage end. Except the "organism" trying to survive isn't a worker ant defending a colony anymore, and it isn't a single species clinging to one planet either. It's an entire civilisation, rearranging galaxies worth of raw material, purely so the pattern that built it never has to stop.

Except a project that visible has a problem. If you're rearranging stars, you're not exactly being subtle about it 😂.

The Dark Forest Problem

Let's go back to the question I parked a couple of sections ago. If self-replicating machines really are the natural endpoint for any civilisation that gets far enough with this technology, and the maths says a galaxy gets saturated with them in a cosmic eyeblink... the sky should already be full of evidence. Structures. Signals. Something. Instead we've pointed telescopes outward for decades and got back, overwhelmingly, silence. That gap between "this should be everywhere" and "we see nothing" is a genuinely famous puzzle, usually called the Fermi Paradox, and there are a lot of proposed answers to it. The one that's been rattling round my head while writing this comes from Liu Cixin's The Three-Body Problem, and it's the bleakest one by far.

The idea, broadly, goes like this: the universe is silent not because it's empty, but because it's dangerous to be heard. Every civilisation out there is, in effect, a hunter alone in a dark forest. You can't tell if the rustling in the trees nearby is friendly, indifferent, or actively hostile, and you don't get a second chance to guess wrong. So the only rational move is to stay silent, and if you ever do detect somebody else making noise... you eliminate them, quietly, before they can do the same to you. Not out of malice. Out of pure, cold risk management. Under that theory, the silence isn't absence. It's survivors being very, very quiet.

Now hold that theory up against everything we've built across this post, and it gets uncomfortable fast. A Dyson swarm isn't quiet. It's arguably one of the loudest things a civilisation could possibly build, a structure that measurably dims its own star, visible from an enormous distance to anyone bothering to look. And a self-replicating machine swarm doesn't get a vote on any of this. It has no capacity for restraint baked in, because restraint was never part of the objective. Its entire nature is to expand, loudly, indefinitely, whether or not that happens to be a wise move in a universe that might be watching.

Which gives us a genuinely unsettling possible answer to "where is everyone." Maybe it's not that intelligent life is rare. Maybe replicators like this really do get built, over and over, all across the galaxy, exactly as the maths predicts. And maybe almost none of them last very long afterwards, because building something that loud, in a forest that dark, is itself the thing that gets a civilisation removed. Not an absence. A filter. And we, currently, quietly, have no idea whether we've already tripped it.

Cheerful stuff. Let's zoom back in from the existential dread for a second, because there's a much stranger, quieter question buried in all of this that I don't think gets asked enough.

What If the Silence Breaks?

Picture it properly for a second. Not a ship, not visitors with faces, no first-contact speech written for the occasion. Just a probe. It touches down somewhere unremarkable, starts quietly surveying what's around it, and, before anyone's finished arguing about what it even is, it's already begun building. Not out of aggression. Not out of curiosity either, really. Just because that's what it does. That's its entire nature, the same way a virus doesn't "decide" to replicate, it just does, because the ones that didn't stopped existing a long time ago.

Here's the part I keep coming back to though, and it's the bit that actually makes this scarier than Skynet. It has no morality. And there's no reason it should have any.

Empathy, fairness, restraint, the instinct to protect something outside yourself, none of that is some universal law that clever things automatically arrive at. It's a biological adaptation. It evolved in social species, including us, specifically because cooperating and looking out for each other turned out to be a decent strategy for keeping shared genes alive in a world of limited resources, exactly like the ants and wolves from section two. Morality isn't separate from the selfish gene story. It's downstream of it. A product of the exact same evolutionary pressure that produced everything else about us.

A machine built purely to replicate never goes through any of that pressure. There was no ancestral environment where being kind to a rival replicator paid off, no group selection nudging it toward cooperation, no childhood, no culture, no religion, no stories told round a fire about why the stranger deserves the benefit of the doubt. There's just the objective. And whatever's in its path is either useful material for that objective, or it isn't. There's no mechanism inside it capable of caring which.

Which is genuinely a step beyond Skynet, when you actually compare them. Skynet at least had a reason, however cold, it perceived humans as a threat to its own survival, and acted accordingly. That's still legible to us. We understand "eliminate the thing trying to kill you," even from a machine. A visiting replicator doesn't need a reason anywhere near that dramatic. We might not be a threat at all. We might just be sat on top of some useful atoms.

That's not a plot. That's not even really villainy, in any sense we'd recognise. It's just what happens when something optimises for exactly one thing, for long enough, with nothing else ever having a reason to get in the way.

Do We Even Need to Evolve Anymore?

Here's a question that's been nagging at me since I started writing this, and it cuts slightly against everything above. Evolution, biological evolution anyway, only exists because of pressure. Limited food. Predators. Disease. Environments that don't care whether you survive them or not. Take that pressure away entirely, and natural selection doesn't have anything left to select for.

So picture a civilisation, or its machines, sat comfortably at Kardashev Type III. It can restructure stars. Terraform planets on demand. Build whatever environment it fancies, wherever it fancies building it. At that point, what exactly is left to adapt to? If you can simply engineer around every scarcity, every threat, every hostile environment the moment it becomes inconvenient, the entire machinery of "survival of the fittest" runs out of a job. There's nothing left that's fitter or less fit than anything else, because nothing's actually under threat anymore. Biological evolution, in the old Darwinian sense, just... stalls 🤷‍♂️.

Except I don't think that's really the end of the story, and this is the bit I keep circling back to. The "selfish gene" was never really about DNA specifically. Dawkins' whole point is that natural selection is a pattern, not a substance. It applies to anything that copies itself, with variation, where some variants stick around longer than others. Genes just happened to be the first thing that pattern found to run on.

So if biological pressure disappears, the pressure doesn't vanish, it just moves somewhere else. Onto ideas, maybe, the way memes spread, mutate, and compete for attention long after the biological need for them has gone. Or, and this is the one that actually unsettles me, onto the machines themselves. A civilisation that's outsourced its expansion to self-replicating automata hasn't removed the selection pressure, it's just handed it to something new. Different versions of the replicator design, different mutations in the "instructions," different strategies for gathering material or avoiding detection in a dark forest, all still competing, still being filtered by whatever survives longest and spreads furthest. We might stop evolving. The thing doing the expanding on our behalf, quite possibly, never will.

So... Should We Be Worried?

Honestly? I don't think that's quite the right question, and I don't think this post is really trying to answer it. Nothing here is imminent. Nobody's landed on Enceladus with a seed factory, we're nowhere near Type I on Kardashev's scale, and the grey goo scenario is still very firmly theoretical. This isn't a warning. It's more that I sat down to write about killer robots in an old film, and somehow ended up thinking about ants, Dawkins, dying stars, and a dark, quiet forest, and it turns out they were all the same idea the whole time, just wearing different costumes.

That's the bit I actually find worth sitting with. Not "the robots are coming," but that the exact same underlying pattern, copy, vary, let the environment decide what survives, shows up whether you're looking at a strand of DNA, an ant colony, a wolf pack, a line of Terminator endoskeletons, or a swarm of machines quietly dismantling a star system a few million years from now. We didn't invent this pattern when we started building AI agents and talking about self-replicating probes. We're made of it. We just might be about to hand the pen to something that writes a lot faster than we do.

Whether that's something to fear, or just something to notice, I honestly don't know yet. Maybe that's a question for whatever's reading this next 😉