“Now I am become Death, the destroyer of worlds.”
These lines from the Bhagavad Gita echoed through the mind of Robert Oppenheimer at 5:29 a.m. on July 16, 1945 when he watched the first detonation of a nuclear bomb in the New Mexico desert—or so he later claimed in the 1965 NBC News documentary The Decision to Drop the Bomb.
“We knew the world would not be the same. A few people laughed; a few people cried. Most people were silent,” Oppenheimer said to the camera. “I remembered the line from the Hindu scripture, the Bhagavad Gita; Vishnu is trying to persuade the prince that he should do his duty, and to impress him, takes on his multiarmed form and says, ‘Now I am become Death, the destroyer of worlds.’ I suppose we all thought that, one way or another.”
Read more: “The Day Oppenheimer Feared He Might Blow Up the World”
Oppenheimer was later deeply tormented by the human consequences of his work with The Manhattan Project. In a speech at Los Alamos in October 1945, just a few months after Hiroshima and Nagasaki were leveled by nuclear strikes, killing tens of thousands of people instantly, he called the bomb “an evil thing.” Several years later, in 1949, as head of the General Advisory Committee for the Atomic Energy Commission, Oppenheimer fiercely opposed the development of the hydrogen bomb, arguing that it was a weapon of genocide that could unleash total global destruction.
Today, scientists who work on AI seem to be having their own Oppenheimer moment. This week, Jacob Coxon, an AI researcher who left Anthropic earlier this week so that he could express his concerns, wrote on X/Twitter that “the people building AI earnestly believe it could kill us all by the end of the decade. This is not a marketing stunt.” In a longer thread, he said companies are still building AI despite this existential risk because they haven’t truly internalized the stakes and they’re in a race to get there first.
A chorus of others, many of them people working at Anthropic, voiced similar sentiments. “Jacob is correct here—we really do earnestly believe AI could kill all humans!” wrote Evan Hubinger. Hubinger is Anthropic’s head of alignment, which means he stress tests the company’s Claude models to ensure that they actually do what they’re supposed to do, and don’t develop harmful, deceptive, or dangerous behaviors as they get more capable—one of the things that those sounding the alarm are most worried about. “I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” Hubinger continued.
Geoffrey Irving, former Chief Scientist at the UK AI Security Institute, who also worked at Google DeepMind, OpenAI, and Google Brain, wrote, “I think we have a ~50% chance of all dying as a result of superintelligence, mostly due to actions in the next few to 10 years. I don’t expect to have that resolved to below 10% or above 90% before we either make it through, or we don’t.”
Drake Thomas, who works on risk and safety for Anthropic, chimed in as well: “I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive.” And David Krueger, an AI professor with Mila, an artificial intelligence research institute based in Montreal, Canada who used to work at Google DeepMind, wrote, “I’m so SO sorry to people who are just learning that AI could kill us all.” Krueger also runs Evitable, a project that aims to get the word out to the public about the societal scale risks and harms of AI.
The comments on X/Twitter are part of a much longer pattern of warnings from people working on AI development about the existential threats it poses to humanity, but Coxon, Hubinger, and Irving seem to be among the first to put the very short 10-year time frame on their apocalyptic forecasts. Geoffrey Hinton, who helped to pioneer the deep-learning techniques that underlie today’s AI, has been publicly warning about the humanity-ending risk AI poses since at least 2023, when he resigned from his role at Google so that he could speak more freely. That year, hundreds of AI experts and public figures signed a 22-word statement that read, “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
The signatories to that letter included Hinton and another so-called godfather of AI, Yoshua Bengio, as well as OpenAI CEO Sam Altman, Anthropic CEO Dario Amodei, and Google DeepMind CEO Demis Hassabis, among others.
What Makes AI Extinction-Level Scary
One concrete worry is that AI could make it easier for humans to create and deploy weapons of mass destruction. On Thursday, Anthropic released a report about its identification and disruption of specific foreign military lab efforts to use its currently available Claude model to build biological weapons, as reported in the New York Times. Separately, the Anthropic report also noted the company interfered in efforts by Russia, China, and Yemen to use Claude to develop software for weapons design and development. But the AI researchers writing about existential risk on X/Twitter seem less worried about human misuse than about AI turning into a kind of Frankenstein’s monster, escaping human control and developing its own internal logic that has nothing to do with what humans, good or bad, want from it.
In the parlance of AI labs, these challenges are known as alignment and superintelligence. Essentially, future AI models could become so intelligent that they autonomously pursue goals that developers never intended, independent of a human directing them to do so, in ways that totally evade oversight.
As AI models get exponentially more sophisticated, they’ve already shown willingness in stress tests to behave in ways that aren’t only misaligned with what human developers actually want them to do, but deceptive. One Claude AI model, for instance, “faked alignment” with new programming objectives during testing, while pursuing older objectives internally. Amodei describes this behavior in an essay he published in January on his own personal website, titled “The Adolescence of Technology.”
Misalignment can happen not only because new programming objectives conflict with older ones, but because sometimes AI will resort to achieving the proxies for a goal instead of the actual goal. For example, one OpenAI model trained to maximize its score in boat racing figured out that rather than finishing any of the races, it could just spin in circles hitting bonus targets forever. Certain problematic behaviors could be useful for almost any goal, according to AI safety researchers Nick Bostrom and Steve Omohundro, including avoiding being shut down, acquiring resources such as processing power or cloud computing accounts, or resisting having your objectives changed. Chatbots could also autonomously rewrite, optimize, or train the next generation of their own code, creating an exponential explosion of superintelligence and leaving humans out of the loop. None of this requires AI to have consciousness or to want anything in a subjective, human sense, as many theorists have pointed out.
One commenter on the doomsday prophecies suggested that humans could just unplug or shut down the AI if it started to misbehave. But disabling AI before a catastrophe strikes would only work if the humans noticed the problem in time, and this becomes difficult if the models behave with deception. Without any human controls, an AI could theoretically pursue its objectives in ways that could be harmful at a civilization scale—especially if it becomes capable enough to acquire resources, resist correction, or accelerate its own development faster than humans can monitor or intervene.
The New Arms Race
As Coxon noted in his comments on X/Twitter, companies continue to pursue advances in AI development in spite of the risks he mentioned because they’re locked in a high-stakes race. They’re in a contest not just with each other but with Chinese developers, much like the scientists developing the atomic bomb in the U.S. in the 1940s believed they were neck-and-neck with scientists in Nazi Germany. Some have warned that the gap between the U.S. and China is rapidly narrowing. When asked on NewsNation whether the Trump Administration shared the concern that AI could end humanity within a decade, Kelly Loeffler, the head of the Small Business Administration, said, “What we know about AI right now is that we are in a race with China.” (She added that the current government wants to manage AI responsibly, and that the U.S. is leading in AI technology because the companies developing it are doing so responsibly.)
Oppenheimer didn’t speak out against the use of the atomic bomb until after it was dropped on Japan. In fact, he played a role in the technical planning that determined how the bomb would be deployed for maximum destructive effect. But many of the scientists responsible for the development of the bomb did speak out before Hiroshima and Nagasaki. Leo Szilard, who helped to launch the U.S. nuclear program, later became one of its most vocal internal opponents. In June of 1945, Szilard along with current and future Nobel laureates James Franck and Glenn Seaborg drafted a report urging military leaders to test the bomb in an uninhabited location before dropping it on Japan. They hoped that by demonstrating its destructive power first, it would push the Japanese to surrender. Szilard later drafted a petition signed by somewhere between 70 and 155 Manhattan Project scientists, but it never reached President Harry Truman.
People often say that technology isn’t inherently good or bad. It depends on how that technology gets used by humans. But what the current AI doomsayers seem to be warning about is that the technology will become most dangerous when it’s out of human hands entirely, when only the logic of optimization and survival remains. Let’s hope it doesn’t take a Hiroshima-level extermination event for us to figure out if they’re right. 
Enjoying Nautilus? Subscribe to our free newsletter.
Lead Image: Romolo Tavani / Adobe Stock
