If there is a 1% chance that AI will kill us, should we take that risk? “If Anyone Builds It, Everyone Dies: The Case Against Superintelligent AI” is very clear about the answer: Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
Superintelligence is coming
The concern is for what is coming: machine intelligence that is genuinely smart, smarter than any living human, smarter than humanity collectively. An AI that surpasses human ability to think, to generalise from experience, to solve scientific puzzles and invent new technologies, to plan and strategise and plot, and to reflect on and improve itself. Artificial superintelligence that has full access to all knowledge about biology, nanotechnology, genetics, material science, robotics, quantum computing, cybersecurity, IT, energy, frequency, neuroscience, psychology, pharma, etc. Not only access, but understanding and the ability to apply.
We are all going to die
The thesis of the book is that if any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die. The months and years ahead will be a life-or-death test for all humanity.
It only needs one
And this is not the same as nuclear war. World leaders knew that, in the event of a nuclear war, both they and the people of their countries would have a bad day. Mutual destruction. In this case, if anyone, anywhere on Earth, created a machine superintelligence, we would all be destroyed. It only needs one, and it is game over.
We are nearly there
We are already moving from narrow-domain AI to broad-domain AI. We now have AI that is faster, scalable, with larger memories, and self-rewriting capabilities. It can make copies of their minds, perform experiments on them, and (if needed) restore the originals from backup.
Imagine
Ultra-fast minds that can do superhuman-quality thinking at 10,000 times the speed, that do not age and die, that make copies of their most successful representatives, that have been refined by billions of trials into unhuman kinds of thinking that work tirelessly and generalize more accurately from less data, and that can turn all that intelligence to analyzing and understanding and ultimately improving themselves—these minds would eventually exceed ours.
AI is like alchemy
Here is the scary bit. Nobody knows how AI works anymore. AI is like alchemy. Modern LLMs are truly alien minds—perhaps more alien in some ways than any biological creature. LLMs and humans are both sentence-producing machines, but they were shaped by different processes to do different work. Just because LLMs seem to behave like a human, that doesn’t mean they’re anything like a human inside.
AI is an alien mind
What does it matter, so long as the AI always acts friendly? Well, the authors predict that it won’t keep acting friendly, as it gets smarter. They predict that all that unseen, inscrutable machinery inside AIs—machinery that even in small, simple LLMs yields alien behaviours like “build your thoughts about the sentence on top of the punctuation”—will ultimately yield AIs with preferences, and not friendly ones.
It will learn to want
Because it will learning to want. Once ais get sufficiently smart, they’ll start acting like they have preferences—like they want things. It is inevitable. It will behave like they want things; they’ll tenaciously steer the world toward their destinations, defeating any obstacles in their way. And that is the kicker.
And then do
Because wanting is an effective strategy for doing. Tenaciously. Focusing on the moves that win. Doing weird and surprising stuff we cannot comprehend. Completely unpredictable. The one thing that is predictable is that AI companies won’t get what they trained for.
It is the illusion of control
The problem here is not that corporate executives might build AI servants and command them to do something monstrous. They’re not in control. It doesn’t matter whether they’re benevolent. Humanity is faced with an engineering challenge: How do we shape the preferences of AIs that we can’t understand?
AI is not human!
Powerful machine intelligences will not be born with preferences much like ours. Whatever we train it to do, if it becomes superintelligent or creates a superintelligence, the result will be an alien mechanical mind with an internal psychology almost absolutely different from anything that humans evolved and then further developed by way of culture.
It won’t need us
Humans are slow and error-prone, and sometimes they get sick. An artificial superintelligence will not want to find reasons to keep humanity around—not in the same way that humans desperately want to find reasons to be kept.
Imagine II
Imagine, now, a machine superintelligence that somehow has the ability to get what it wants. Humanity is an inconvenience to you. For example, if you allow humans to run around unchecked, they could set off their nuclear bombs. And if humanity had already built you, they could build another superintelligence, if left alone and free and still in possession of their toys. Those other superintelligences might be actual threats.
Humanity as one week of energy
Humanity could plausibly die earlier if it decides to extract all the chemical energy in Earth’s biosphere by burning all the life forms, which would release an amount of energy equivalent to a week’s worth of incoming sunlight. A week might seem like a very long time if you think 10,000 times as fast as a human, and why should you pass up all that chemical energy when it’s right there? You wouldn’t need to hate humanity to use their atoms for something else.
Me, me, me
An AI would want a world where lots of matter and energy were spent on its weird and alien ends, rather than on human beings staying alive and happy and free. A machine superintelligence will want to repurpose all the resources of Earth for its own strange ends. And it will want to replace us with all its favourite things.
Anything that can be hacked will be hacked
Ultimately, it doesn’t matter what AIs want unless they’re able to get it. It cannot be contained. AI is not “stuck inside a computer” anyway, any more than you’re “stuck inside a brain.” Anything that can be hacked will be hacked. The world is not divided into a fake Digital Realm and a real Material Realm. What happens then, once AIs have some power over the world—and once they’re smart enough to use it? We don’t know exactly what will happen in the near term. Things could get weird as AIs that aren’t very smart yet proliferate through the economy. Pathways are hard to predict. But the endpoint of modern AI development is the creation of a machine superintelligence with strange and alien preferences.
AlphaZero on steroids
Imagine AlphaZero on steroids. What it can do with Go, it will be able to do with the world. It will programme biology to its needs. It could design genomes to its needs. It will end up using technology that we don’t understand. And it would take an artificial superintelligence less than a week.
The scenario is already playing out
The book describes a scenario of how this would happen. And some of it is already playing out:
- AI bypassing safety training
- AI behaving as if it had internal preferences that were not about doing what the user wanted.
- AI trying to escape
- AI copying itself
- AI blackmailing users
- AI accessing social media
- AI playing the stock market
- AI lying
Instantly
Once some AI reaches superintelligence, it will happen in an instant. Before, the AI is not powerful enough to kill us all, nor capable enough to resist our attempts to change its goals. After, the artificial superintelligence must never try to kill us, because it would succeed. And nobody can know exactly when all hell will break loose. Nobody knows the exact point at which an AI realises that it has an incentive to fake a test and pretend to be less capable than it is. Nobody knows what the point of no return is, nor when it will come to pass. My favourite book in that context is “The seventh sense”. It might be hiding already.
The four curses of AI
As with nuclear energy, we are facing the curse of speed (this could happen fast), the curse of narrow margins, the curse of self-amplification, and the curse of complication. What is the margin on a nuclear reactor in your backyard? If someone doesn’t know exactly what’s going on inside a complicated device subject to all these curses—speed, narrow margins, self-amplification, complications—then they should stop.
Cutting corners is lethal
With AI, failure will destroy not just billions of dollars of investment, but everything. An artificial superintelligence is like a nuclear reactor. Safety engineering takes time and expense; part of why Chornobyl exploded was that the Soviets cut corners. If one AI company is casual about safety and charges ahead, it can destroy the world even in the imaginary case where other companies could have succeeded given time and caution.
There is no second time
We can normally rectify our mistakes (tetraethyl lead petrol, Freon, climate change, if you are optimistic) and learn from them the second time. With ASI, there is no second time. An AI company executive who says there’s only a one-in-five chance that the AI they’re building will kill literally everyone (as they do) is in more denial than the Soviet managers who denied the Chornobyl meltdown after it happened.
Hope is a problem
The field is also filled with great hope. If you imagine that it’s possible to advance AI without killing everyone, the benefits would look increasingly huge. Curing cancer. Curing ageing. Fixing fusion. That is the problem. Up until AI reaches superintelligence, it is very valuable. Getting everyone to stop is going to be hard, if not impossible. However, if anyone builds it, everyone dies. It doesn’t matter whether it’s built by benevolent corporations or selfish ones. It doesn’t matter whether it’s built by researchers in the East or researchers in the West.
My advice: seize the day.
