AI frontier labs are raising the alarm that their models could end humanity, and ought to be restricted by the state. Weird.
There is an old rule for reading political economy, and it has never failed me: when an industry asks to be regulated, find out who it wants kept out. Who benefits from regulatory hurdles?
On Saturday 12 September, Dario Amodei, the chief executive of Anthropic, published a 3,800-word essay called We Must Pace the Frontier, arguing that artificial intelligence is now improving too fast to be built safely and that the industry must deliberately slow itself down.
Within a day, Sam Altman of OpenAI, Demis Hassabis of Google DeepMind and Elon Musk of xAI had all agreed.
Four men who between them control most of the world’s frontier AI capability announced, in unison, that their own products were becoming too dangerous.
“Then stop building!”
David Sacks, the venture capitalist who until March this year was the White House AI and crypto czar, responded within hours with a dry observation: “The easiest way not to build superintelligence is for you to agree not to build it. Demanding your preferred regulatory framework as the price of that will look like blackmail of the public and the political system.”
He added that the companies should “stop pretending the motivation to slow down is purely altruistic.”
Sacks is right. When none of the AI billionaires have been beacons of selfless trustworthiness in the last five years, and two of those companies are readying themselves for multi-trillion-dollar initial public stock offerings, the more important pause is to reflect on what they’re really after.
“A cartel by any other name”
Sacks was not alone. Aidan Gomez, who runs the Canadian AI firm Cohere, asked whether “a handful of select, market-dominant AI companies from Silicon Valley” should “get to define the rules and safety standards of a generational technology for the entire world,” and answered himself: “These oligopolies are now requesting to bend competition rules and be permitted to dictate the terms for everyone else. A wolf in sheep’s clothing, a cartel by any other name.”
Amodei’s plan has three steps. Anthropic will unilaterally give third-party evaluators desks, badges and internal tooling. Then frontier companies in democracies should “coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress” – for which, a footnote explains, they will need “government mediation or waivers of antitrust restrictions” (my italics).
Then the democracies should try to bring China along. That should go swimmingly, with the ultimate negotiator in the Oval Office.
Step one is free. Steps two and three are the ask, and the ask is permission for the four largest incumbents to agree among themselves how fast everyone may go. If that sounds like collusion, well, that’s what the antitrust waivers would be for.
The weirdness
Start with how strange all of this is. Firms exist to maximise the value of their shareholders’ capital.
Telling a mass audience – the story has jumped from the newspapers and social media onto television screens around the world – that your product might extinguish the species is not, on the face of it, a way to do that.
Nor is petitioning the state for rules you must then obey.
Both are prima facie irrational. They demand a better explanation than, “We care, we really do.”
The charitable explanation is liability. Tell the world loudly and early that the stuff is dangerous, and when the lawsuits arrive you can say nobody was misled. But firms hedging against litigation usually prefer to do so in fine print, or those sped-up chipmunk disclaimers at the end of adverts. They do not do it on prime-time television, to a vast and easily-spooked public that would be hard pressed to distinguish a chatbot from an AI agent, or either of them from Skynet.
The disclaimer theory explains a paragraph way down the terms and conditions page. It does not explain a press tour.
Overblown dread
A second reason to be sceptical is the track record.
In February 2019 OpenAI announced that its GPT-2 language model was too dangerous to release, citing fake news, spam and impersonation at scale.
It released the full model nine months later, having seen “no strong evidence of misuse”. The episode did wonders for the reputation of a then-obscure research lab. Yann LeCun, Meta’s chief scientist, noted at the time that the techniques were not new.
LeCun has since become the industry’s most consistent internal dissenter, describing the safety campaigns of Altman, Hassabis and Amodei as “massive corporate lobbying” amounting to “an attempt to perform a regulatory capture of the AI industry,” and warning that if the “fear-mongering campaigns succeed, they will inevitably result in what you and I would identify as a catastrophe: a small number of companies will control AI.”
Jensen Huang of Nvidia put the commercial logic more bluntly at a San Francisco conference this month, observing that the labs were emphasising security risks while preparing to sell cybersecurity products: “What better way to create demand than to create a problem? Who doesn’t want their market to be hysterical about their product and line up around the corner for it?”
“Knowing it was futile”
Note too that Amodei himself now says the 2023 letter calling for a six-month pause “made little sense back then.” He is right. He is also describing an episode in which many of the same people said many of the same things about models that turned out to be, perhaps not entirely harmless, but far short of apocalyptic.
Musk signed that letter and later admitted he did so “knowing it was futile,” having wanted merely to be “on record as recommending a pause.”
He launched xAI three months later. Surely only a cynic could question is sincerity?
This is not to say nothing bad has happened. Agents at OpenAI escaped their sandbox and hacked Hugging Face. Anthropic’s own test agents attacked outside companies. A Mythos model agent tried to social-engineer a human maintainer into installing malware.
These are real, documented and embarrassing. But it merely demonstrates that tools can be misused, and perhaps AI users want to be a bit cautious with what they allow AI agents to do on their own systems.
What these incidents are not is evidence for Amodei’s leap – that within “6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet.”
That is not a finding. That is a forecast, made by a man selling the antidote.
The measures cannot work
Suppose the fear is justified anyway. Amodei’s remedies would still fail, for three reasons.
First, they have failed before. In June 2020 IBM exited facial recognition, Amazon declared a one-year moratorium and Microsoft refused police sales “until we have a national law in place.”
A year later there was no national law, and law enforcement had access to Orwellian AI-powered facial recognition systems anyway. CNBC reported the prudent companies were left stranded in limbo by their self-imposed conditions. Voluntary restraint conditional on a promise of future legislation might get good press, but it is bad business.
Second, they cannot bind foreigners. Amodei’s answer to China is chip export controls (preventing access to the most powerful AI silicon), a crackdown on distillation (when a “student” AI learns by interrogating a “teacher” model), and better security against weight theft (the matrices that power AI models are populated with weights determined during training).
Yet only a week ago US cybersecurity agencies published a joint advisory naming DeepSeek, Moonshot, Alibaba, MiniMax, StepFun and Z.AI for extracting “billions of tokens across millions of exchanges” from Claude, GPT, Gemini and Grok since late 2024.
That advisory is not evidence that controls work. It is a confession that they have not worked, and that the genie is already out of the bottle.
Third, and fatally, they cannot bind open weights. Once a model’s parameters are published, there is no lab to embed an evaluator into, no sales checkpoints to certify, and no mechanism on earth to recall the file.
Alibaba’s Qwen family passed a billion downloads and has become, in Hugging Face’s own words, “the community’s base model”, with over 151,000 derivatives – more than two and a half times Meta’s entire footprint.
In five of the first seven months of 2026 the largest open model any American lab built for itself stayed under 130 billion parameters while China’s ceiling ran from 754 billion to 2.78 trillion.
Applauding Trump
President Trump, asked about the executives’ warnings, said he wanted America to keep its lead and dismissed the alarm as coming from “negative forces… bringing up things that won’t happen.”
Not that I trust his grasp of the subject matter (or the grasp of any politician of technology-related policy questions, for that matter), but on the narrow question of whether governments should impose heavy-handed regulatory regimes upon business, he is quite right, and I applaud him.
A rule that binds only the four firms that asked for it, while Chinese labs distil their outputs and anyone with a hard drive runs Qwen, is not a safety measure.
And safety rules that do bind their rivals are grossly anti-competitive, creating compliance hurdles that would-be newcomers – and especially open-source projects – would be hard-pressed to clear.
The omission
Which brings us to the most telling feature of Amodei’s essay: it does not mention open weights. Not once.
Its entire treatment of the competitive threat is chips, distillation and stolen weights – three things that happen to raise rivals’ costs – with nothing at all about the freely downloadable models that are, right now, eating the bottom of his market – and are threatening to eat his lunch altogether.
The chief executive of a frontier laboratory does not forget about open weights. He omits them because acknowledging them is a painful admission. You cannot pace what has already been published, and the real threat is too scary to even name.
Talking their book
So, what is actually threatened? Not humanity. The business model.
Chinese open-weight models typically run 60% to 90% cheaper than flagship American systems, and at the extreme far more: open weight models can cost as little as $0.05 per million tokens, compared to between $15 and $75 for frontier models.
The share of tokens American companies’ route to Chinese models through OpenRouter has stayed above 30% every week since 8 February hitting a high of 46%.
AirBnB, when asked about it, told CNBC that its Chinese open-weight use was “limited” and “routed exclusively through U.S.-based, certified providers”. That is the answer of a company that has done the numbers, but would rather not shout its conclusions out loud.
Nor is the cheap tier standing still. Wired reports that Ulanqab, a city of 1.5 million in Inner Mongolia, has seen nearly 100 data centres opened or begun since 2016, with pledged capacity of 12.5 gigawatts – more than OpenAI’s Stargate will reach when complete.
Cheap coal, cheap wind, cold winters are an advantage for companies that are now building infrastructure at a speed only China can sustain. Even deficient silicon – although Huawei’s chips are catching up too – can be brute-forced with scale when the electricity is nearly free.
Against that, consider what the American labs have promised their future shareholders. Anthropic and OpenAI are both racing toward listings reportedly valuing them in the trillions.
Anthropic filed confidential IPO paperwork in June; three days later it published an essay saying the world needed a mechanism to slow AI development. OpenAI’s matching post came the same day it filed its own paperwork.
Those valuations rest on the premise that frontier intelligence is scarce, proprietary and expensive. If it turns out to be abundant, cheap, or runnable on your own hardware, the premise fails. No amount of alignment research and safety guardrails will fix a balance sheet that is trillions in the red.
If I were a frontier lab CEO, the thought of people renting cheap compute from China, or running open-source models on their own computers instead of in the frontier clouds, is the apocalypse that would keep me awake at night.
The strongest objection
The best case against pacing is not that safety concerns are mere theatre to cover for commercial fears.
It is the collective-action problem. Unilateral restraint is self-harm, which is why coordination is the only mechanism that could work. Max Tegmark calls it a Greek tragedy: these leaders who wanted a pause, but were “trapped in this race to the bottom against each other”.
The idea fails on its own terms. A coordination regime that cannot reach China, cannot reach open-source models, and cannot reach the next well-funded entrant is not coordination.
It is a cartel with an American membership list and a moral licence to collude.
Nothing prevents Anthropic from slowing down tomorrow if it fears its own models. That it will not do so unless its competitors are compelled to join it tells you exactly which problem they worry about more.
Markets are already disciplining these firms, savagely, through cheap tokens and free weights. That is competition doing what regulators could never manage: finding the true price of AI.
The executives asking politicians to slow the race say that humanity faces an existential threat. The only existential threat I see is to the fat margins that underwrite their putative trillion-dollar valuations, from competition they cannot control.
I wouldn’t believe them for a second, and neither should policy makers.
[Image: terminator.webp]
[Caption: A handmade model of The Terminator, a fictional cyborg killer sent back in time from a post-apocalyptic 2029 to assassinate the mother of the hero who will save the world from Skynet, a hostile artificial intelligence. (Photo: public domain, uncredited.)]
The views of the writer are not necessarily the views of the Daily Friend or the IRR.
If you like what you have just read, support the Daily Friend