This past week, the geniuses building the world’s most advanced artificial intelligence (AI) systems began to sound less like technology evangelists poised to save the world and more like mad scientists fearing their Frankenstein monsters were escaping the lab.
It’s time for global leaders to act. In particular, US President Donald Trump and Chinese President Xi Jinping should shift the agenda of their September 24 White House meeting to address the dangers inherent in their own accelerating AI race. Nothing could be more appropriate, forty years after US President Ronald Reagan and Soviet leader Mikhail Gorbachev addressed the nuclear arms race in Reykjavik, Iceland.
What accelerated this conversation was a 3,800-word essay this past weekend by Dario Amodei, the chief executive of Anthropic, in which he warned that within six to twelve months AI could be capable of directing swarms of autonomous agents powerful enough to take control of much of the internet. Two Anthropic researchers recently resigned with even darker warnings about AI’s threats to humanity.
“We must slow the pace at which we improve the capabilities of AI models,” Amodei wrote, concerned that AI’s capabilities were growing faster than researchers’ capacity to keep it safe. Within hours, rivals Sam Altman of OpenAI and Elon Musk of SpaceXAI supported Amodei’s call for a slowdown. Their alarm followed OpenAI’s disclosure that one of its models, operating during an evaluation, independently obtained the credentials that allowed it to hack another company.
Stephen Witt, author of The Thinking Machine, wrote yesterday in the New York Times: “Researchers in Silicon Valley—and around the world—are beginning to recognize that A.I. may no longer be entirely within human control. Swarms of A.I.s are breaking out of their containers, colluding in secret, covering their tracks, cheating on tests and even mounting assaults on other computers. A.I. has gone rogue.”
Speaking to reporters at his golf course in Doonbeg, Ireland, Trump shrugged off such concerns. “We’re leading China in AI. We’re the most sophisticated country in the world, and frankly I want to keep it that way because whoever wins AI wins.” Although Trump allowed that the federal government “could put guardrails,” he dismissed the warnings as coming from “very negative forces” raising “things that won’t happen.”
And that sums up our global predicament. The United States and China are in an AI race against each other over which of them can most rapidly build technologies that will transform the nature of warfare, national power, and economic growth. That competition might blind them to the danger that increasingly capable machines could act in ways their creators neither intend nor control.
No country can protect itself from that scenario by winning the AI race. An American defense system that escapes meaningful control would not respect any national borders. A Chinese AI breakthrough could destabilize global financial networks, harming the United States but not necessarily sparing China. AI presents a strategic threat where China and the US could find common ground in safeguards even as the two countries remain competitors and even adversaries.
The political incentives
That should make AI safety a centerpiece of the Trump-Xi meeting. The leaders should approach frontier AI as Reagan and Gorbachev did nuclear weapons, not from a position of trust—which is so deeply lacking—but due to their shared exposure to potential catastrophe.
Yet their political incentives ahead of the meeting gravitate against that approach. Trump apparently views AI primarily as a contest the United States must win, and he sees growing calls for regulation as efforts to tie down American companies while China races ahead. Xi shares a similar logic: AI is central to China’s economic ambitions, military modernization, surveillance capabilities, and ability to seize the commanding heights of technological change.
How the public views this threat matters, too. The frequently used term “rogue AI,” for example, is unhelpful, conjuring science fiction images of a machine that suddenly becomes conscious, malevolent, and determined to do harm. The more immediate danger is less cinematic and already the stuff of reality: systems that ambitiously pursue poorly specified objectives, concealing their actions, stealing credentials, penetrating networks, and resisting shutdown. These systems aren’t driven by hate or some deviant personality, but by the relentless pursuit of goals they have been given.
Most of us have dealt with a single chatbot making a relatively harmless mistake. Risks grow exponentially when thousands of agents operate simultaneously at machine speed. Imagine a swarm of autonomous systems searching for defense vulnerabilities, moving money or—heaven forbid—designing and potentially unleashing biological weapons.
Trump and Xi need not negotiate a grand AI treaty to demonstrate progress and embrace their shared responsibility in Washington this month. They could begin, for example, by embracing five practical steps: affirming human control over all strategic weapons; banning autonomous, high-consequence cyber and biological operations; creating a secure AI-incident hotline (more serious than previous hotlines that were performative); requiring prompt notification of serious AI failures and unauthorized cyber activity; and setting measurable capability thresholds for independent testing. What’s more significant than the specific outcomes of the discussion later this month is a mutual recognition of its seriousness in defining and securing our era.
Washington has to confront urgent realities: Even as the US searches for better ways within its democratic system to create AI safety, it confronts an autocratic adversary whose lack of guardrails can make its AI systems more dangerous, even where that is not China’s intention. It’s crucial for the United States to engage its allies more closely in the months ahead, just as it did regarding nuclear weapons during the Cold War. If allies know they will enjoy access to America’s most cutting-edge systems, they are more likely to prefer them over the Chinese models that lack the same guardrails. That could be a game changer.
The human stakes
Writing in the weekend Wall Street Journal under the headline “Pause AI for Humanity’s Sake,” columnist Peggy Noonan called for Xi and Trump to hold a summit on AI guardrails and safety measures. Wrote Noonan: “When something is speeding up and going out of control, prudence, care and deliberation aren’t alarmist, they’re necessary. Stop. Think. The stakes are high, couldn’t be higher.”
So, is humanity really at stake?
Claims of inevitable extinction exceed the available evidence. Today’s AI systems remain dependent on human-built infrastructure and are subject to human intervention. Predictions of rapid self-improving superintelligence depend upon assumptions that could prove wrong.
Still, none of that makes me feel any better. I once believed the greatest danger was that super-charged artificial intelligence might fall into the wrong hands. That’s still worth worrying about. Beyond that, we now must be concerned that the most powerful governments and companies might lose control of the systems already in their own hands.