China Hawks are Manufacturing an AI Arms Race (Now with a Reply from Gwern)
An influential congressional commission is calling for a militarized race to build superintelligent AI based on threadbare evidence
[EDIT: the inimitable Gwern left fascinating comment on this post on LW, I’ve repasted in it in full at the bottom of the piece.]
The US-China AI rivalry is entering a dangerous new phase.
Earlier today, the US-China Economic and Security Review Commission (USCC) released its annual report, with the following as its top recommendation:
Congress establish and fund a Manhattan Project-like program dedicated to racing to and acquiring an Artificial General Intelligence (AGI) capability. AGI is generally defined as systems that are as good as or better than human capabilities across all cognitive domains and would surpass the sharpest human minds at every task.
As someone observed on X, it’s telling that they didn’t call it an “Apollo Project.”
One of the USCC Commissioners, Jacob Helberg, tells Reuters that “China is racing towards AGI ... It's critical that we take them extremely seriously.”
But is China actually racing towards AGI? Big, if true!
The report clocks in at a cool 793 pages with 344 endnotes. Despite this length, there are only a handful of mentions of AGI, and all of them are in the sections recommending that the US race to build it.
In other words, there is no evidence in the report to support Helberg’s claim that "China is racing towards AGI.”
Nonetheless, his quote goes unchallenged into the 300-word Reuters story, which will be read far more than the 800-page document. It has the added gravitas of coming from one of the commissioners behind such a gargantuan report.
I’m not asserting that China is definitively NOT rushing to build AGI. But if there were solid evidence behind Helberg’s claim, why didn’t it make it into the report?
Helberg has not replied to a request for comment.
As the report notes, the CCP has long expressed a desire to lead the world in AI development. But that’s not the same thing as deliberately trying to build AGI, which could have profoundly destabilizing effects, even if we had a surefire way of aligning such a system with its creators interests (we don’t).
I was hoping the report would marshal the strongest evidence for Helberg’s claim, but I found remarkably little analysis about China’s AI intentions, beyond summaries of the country’s desire to develop the industry to be robust to American containment efforts.
What China has said about AI
In July 2017, China’s State Council published an important document called “A Next Generation Artificial Intelligence Development Plan,” which was translated by the New America think tank. The plan serves as blueprint for the country’s AI industrial policy and includes the often-quoted goal of leading the world in AI by 2030.
Does this mean creating a digital superintelligence that will permanently alter the global balance of power?
Not exactly.
Here’s how the goal is introduced:
Third, by 2030, China’s AI theories, technologies, and applications should achieve world-leading levels, making China the world’s primary AI innovation center, achieving visible results in intelligent economy and intelligent society applications, and laying an important foundation for becoming a leading innovation-style nation and an economic power.
The document includes targets for an AI industry valued at 1 trillion RMB by 2030 (about $190 billion in today's dollars). For context, Statista projects the 2030 global AI market to be $827 billion, with the US at $224 billion and China at $155 billion.
In other words, China's 2030 targets from 2017 are only a bit more optimistic than current market projections — and would actually fall short of global dominance. These numbers suggest normal industrial growth ambitions, not the kind of revolutionary technological breakthrough implied by AGI. (Some AI forecasters think AGI could drive annual per capita GDP growth well above 100%.)
Also in the plan is the aspiration that China “will have constructed more comprehensive AI laws and regulations, and an ethical norms and policy system.”
A few days ago, Joe Biden and Xi Jinping met in Lima, Peru, where the Chinese president reportedly called for “more dialogue and cooperation” and discussed AI as a “global challenge” in the same vein as climate change.
And in July, the CCP released a document that China AI expert Matt Sheehan said is “the clearest indication we've seen that concerns about AI safety have reached top CCP leadership, and that they intend to take some action on this.” Sheehan has previously written that “Beijing is leading the way in AI regulation,” something Anthropic’s policy chief has also acknowledged.
Obviously, we should take all of this with a grain of salt. World leaders have an incentive to exaggerate their willingness to play ball and act in the global interest, and the significance of Chinese AI regulations isn’t totally clear. But policymakers should at least be aware of the large gap between what China says and does when it comes to AI, and what hawks assert the country is doing or planning (especially when they don’t cite evidence).
Only one superpower has a government commission publicly calling for a militarized race to build superintelligent AI (with no plan for how to control it), and it’s not China.
Revealing technical errors
There are also some indications that the report authors were a bit out of their depths when it comes to AI.
The report repeatedly misidentifies basic technical concepts. It refers to “ChatGPT-3” multiple times, despite no such product existing — ChatGPT launched using GPT-3.5, an improved version of GPT-3. When comparing model performance, the authors confuse ChatGPT (an interface) with the underlying models like GPT-3.5 and GPT-4. These aren't just semantic distinctions when you're explicitly comparing the capabilities of different AI systems.
The confusion runs deeper. The report claims “OpenAI, a closed model, cut off China's access to its services” — but OpenAI, you might realize, is a company, not a model. It also states that “Generative AI models can transmit algorithms into text, images, audio, video, and code.” This appears to be a garbled paraphrase of a McKinsey definition (itself not particularly precise) about AI generating different types of content.
These may seem like nitpicks, but they reveal a concerning lack of technical literacy in a report meant to guide national AI policy. And speaking as someone who worked at McKinsey, it's not where I'd go for technical definitions of AI.
Most tellingly, the definition they offer for AGI has problems that don’t require any technical expertise to catch:
AGI is generally defined as systems that are as good as or better than human capabilities across all cognitive domains and would surpass the sharpest human minds at every task.
Is AGI just something that is “as good” as humans or something that “surpasses” the smartest of us? This isn’t some obscure definition buried deep in the report. It’s literally the second sentence in their top recommendation. It’s also the goal the authors think the US should mobilize a wartime effort to meet. Taken as written, it’s not clear what it would even mean to achieve it! (Setting aside the usual difficulty of actually defining and measuring AGI.)
Conclusion
We’ve seen this all before. The most hawkish voices are amplified and skeptics are iced out. Evidence-free claims about adversary capabilities drive policy, while contrary intelligence is buried or ignored.
In the late 1950s, Defense Department officials and hawkish politicians warned of a dangerous 'missile gap' with the Soviet Union. The claim that the Soviets had more nuclear missiles than the US helped Kennedy win the presidency and justified a massive military buildup. There was just one problem: it wasn't true. New intelligence showed the Soviets had just four ICBMs when the US had dozens.
Now we're watching the birth of a similar narrative. (In some cases, the parallels are a little too on the nose: OpenAI’s new chief lobbyist, Chris Lehane, argued last week at a prestigious DC think tank that the US is facing a “compute gap.”)
The fear of a nefarious and mysterious other is the ultimate justification to cut any corner and race ahead without a real plan. We narrowly averted catastrophe in the first Cold War. We may not be so lucky if we incite a second.
Appendix: Gwern weighs in
Pasted in full from Gwern’s comment on the LW version of this post:
“Also worth noting is Steve Hsu's recent discussion of his meetings with China VC, government, researchers etc. reporting from on the ground in Shanghai and Beijing etc: https://www.manifold1.com/episodes/letter-from-shanghai-reflections-on-china-in-2024-73/transcript
Hsu is a long-time China hawk and has been talking up the scientific & technological capabilities of the CCP for a long time, saying they were going to surpass the West any moment now, so I found this interesting when Hsu explains that:
the scientific culture of China is 'mafia' like (Hsu's term, not mine) and focused on legible easily-cited incremental research, and is against making any daring research leaps or controversial breakthroughs...
but is capable of extremely high quality world-class followup and large scientific investments given a clear objective target and government marching orders
there is no interest or investment in an AI arms race, in part because of a "quiet confidence" (ie. apathy/laying-flat) that if anything important happens, fast-follower China can just catch up a few years later and win the real race. They just aren't doing it. There is no Chinese Manhattan Project. There is no race. They aren't dumping the money into it, and other things, like chips and Taiwan and demographics, are the big concerns which have the focus from the top of the government, and no one is interested in sticking their necks out for wacky things like 'spending a billion dollars on a single training run' without explicit enthusiastic endorsement from the very top.
Let the crazy Americans with their fantasies of AGI in a few years race ahead and knock themselves out, and China will stroll along, and scoop up the results, and scale it all out cost-effectively and outcompete any Western AGI-related stuff (ie. be the BYD to the Tesla). The Westerners may make the history books, but the Chinese will make the huge bucks.
So, this raises an important question for the arms race people: if you believe it's OK to race, because even if your race winds up creating the very race you claimed you were trying to avoid, you are still going to beat China to AGI (which is highly plausible, inasmuch as it is easy to win a race when only one side is racing), and you have AGI a year (or two at the most) before China and you supposedly "win"... Then what?
race to AGI and win
trigger a bunch of other countries racing to their own AGI (now that they know it's doable, increasingly much about how to do it, can borrow/steal/imitate the first AGI, and have to do so "before it's too late")
???
profit!
What does winning look like? What do you do next? How do you "bury the body"? You get AGI and you show it off publicly, Xi blows his stack as he realizes how badly he screwed up strategically and declares a national emergency and the CCP starts racing towards its own AGI in a year, and... then what? What do you do in this 1 year period, while you still enjoy AGI supremacy? You have millions of AGIs which can do... stuff. What is this stuff? Are you going to start massive weaponized hacking to subvert CCP AI programs as much as possible short of nuclear war? Lobby the UN to ban rival AGIs and approve US carrier group air strikes on the Chinese mainland? License it to the CCP to buy them off? Just... do nothing and enjoy 10%+ GDP growth for one year before the rival CCP AGIs all start getting deployed? Do you have any idea at all? If you don't, what is the point of 'winning the race'?
(This is a question the leaders of the Manhattan Project should have been asking themselves when it became obvious that there were no genuine rival projects in Japan or Germany, and the original "we have to beat Hitler to the bomb" rationale had become totally irrelevant and indeed, an outright propaganda lie. The US got The Bomb, immediately ensuring that everyone else would be interested in getting the bomb, particularly the USSR, in the foreseeable future... and then what? Then what? "I'll ask the AGIs for an idea how to get us out of this mess" is an unserious response, and it is not a plan if all of the remaining viable plans the AGIs could implement are one of those previous plans which you are unwilling to execute - similar to how 'nuke Moscow before noon today' was a viable plan to maintain nuclear supremacy, but wasn't going to happen, and it would have been better to not put yourself in that position in the first place.)”