AI development may be slowed, but economic rivalry, geopolitical competition, and open-source proliferation make stopping it nearly impossible worldwide.
Recently OpenAI announced that during testing an AI agent went rogue and hacked another startup. To be clear, it was not directed or allowed to do that, but it chose to and figured out how to do so anyway. “We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities,” said OpenAI. Just two days earlier, they said, “Previous models, when they hit sandboxing or environmental constraints, would simply stop and return to the user. This model often kept trying, including by looking for ways to act outside its sandbox.” Clever girl.
Shortly before this article was published, an open letter was released called “Pacing the Frontier.” The official statement is, “We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” It should be noted, for reasons we’ll get into below, that this open letter is to the U.S. government.
Is such a pause feasible, even assuming that it's desirable (which is outside the scope of this article)? Society is asking: What are the risks, and can they be stopped? The risks from AI are high. The reality is that AI progress and many of its associated risks cannot be easily stopped; at best, AI progress can be slowed, but even that is limited. This runaway train has no brakes.
Risk is a general term. There’s no single risk to your home. Fires, floods, a neighbor's tree crashing through the wall, and robbery all have different probabilities and impacts. Even robbery has different levels, e.g., a smash-and-grab thief looking for a quick score versus a professional thief who plans a high-end burglary and will defeat advanced security measures present different risk and impact levels. We must be equally descriptive about risks from AI.
In this article I’m not talking about a Terminator-style threat to human life or AI controlling weapons systems. That may come in the future, but we’re a while away from it. We'll consider the following four cases.
The first is directed human hacking. This is nothing new, it’s simply that AI is a hacking supertool. The big advance here is that models like Anthropic’s Mythos can now find zero-day vulnerabilities. They have traditionally been hard to find and required advanced skill; now anyone can find and exploit them. The frequency and scale of hacking will increase.
Second is misalignment. Many people these days talk about paperclip maximization (more generally, instrumental convergence). In 2003, rephrasing Marvin Minsky’s “Riemann hypothesis catastrophe,” Nick Bostrom conjectured that an AI told to produce as many paperclips as possible could destroy the human race. It may not be evil, but to meet its goal it would slowly take over all production in the world, converting all materials, factories, and land into producing more and more paperclips. More generally, misaligned AI may not destroy the world, but it will waste time and maybe do some damage (e.g., delete some critical files to free up space on a hard drive).
The above cases primarily create direct harm, intentional and inadvertent, respectively. Generative AI can also lead to broader systemic harm through widespread use and secondary effects.
The third risk is overreliance and secondary effects. Many people are using AI for therapy, but studies have shown that AI’s sycophancy can decrease prosocial behaviors, that AI often provides bad medical advice, and that its use regularly leads to ethical violations. Then there is the risk that society will lose basic skills because they have been outsourced to AI, a process commonly referred to as “deskilling.” This can involve both mechanical skills (e.g., daily tasks at home and work, such as planning a trip or conducting trend analysis) and social skills. Risks of desocialization, poor human-to-human interactions, and increased loneliness are all risks. (In 2001, Futurama’s S3E15 viewers were warned about some of these risks in the “I Dated a Robot!” PSA.)
Finally, there’s the tax or friction that society will pay from excess content. The percentage of AI-generated content is increasing. (Web searches provided various numbers from different sources. I didn’t feel comfortable enough with any to cite them directly here, but all agree the percentage is increasing.) More and more humans (or their AI proxies) must determine what is real and what is fake; this includes images, videos, articles, scientific papers, phone calls from other people, etc. It adds a time and/or financial burden to all content consumption and related decision-making. I go into this in more detail in “The Shift to a Zero Trust Society.” (For the record, I write all my articles myself but use AI for editing and to create the articles’ images and social media posts.)
I’m intentionally not covering risks to the labor market for two reasons. First, I’ve written about this extensively before (see “No, AI Isn’t Going to Kill You, but It Will Cause Social Unrest - Part 1,” “No, AI Isn’t Going to Kill You, but It Will Cause Social Unrest - Part 2,” “How to Know If Your Job Is Safe from AI — Part 1: What History Shows Us About Job Loss and Job Growth,” and “How to Know If Your Job Is Safe from AI — Part 2: The Economic Drivers of Your Job”). Second, this is the easiest of these risks to mitigate through legislative action, although “easiest” is very much a relative term. (See the referenced articles for a deeper discussion.)
These risks differ in probability and impact, at both the individual and societal levels. Likewise, their mechanisms, methods of prevention, and remedies vary. What they have in common is that economic and political forces all drive toward the rapid adoption of AI, even with these associated risks.
Game theory can be used to describe how companies approach innovation in this area. General generative AI (including LLMs), as well as artificial general intelligence (AGI), represents a winner-take-all race. In the 1990s there were over a dozen search engines. Today Google has a little over 90% of the search engine market, a position it’s held for well over a decade. This position drives a majority of its revenue. A handful of AI companies are likely to own most of the AI market. With trillions of dollars at stake, everyone wants to be first across the finish line. This is why we see companies throwing hundreds of billions of dollars at the problem. The first-place winner gets trillions, the second place gets a set of steak knives, the third prize is you’re fired.
In corporate competition there is a referee: the government. However, the referee only has authority in his league. OpenAI, Anthropic, Google, etc., are all based in the US and under US jurisdiction. (This is why the open letter, signed by employees of US-based companies, addressed it to the US government.) The EU and China are entirely separate. Even if the EU tends to collaborate with the US, China does not. (Russia probably would also be another independent arena, but it’s spending much of its already-constrained GDP on the foolhardy war on Ukraine; however, in the future it could redirect funds to AI R&D.)
Similarly, governments have their own winner-take-all game. One serious concern is that both offensive and defensive weapons systems in the future will be AI-controlled. If one side is even slightly, but firmly, superior to the other, it will win a majority of the battles. Even if it’s not a kinetic war, cyberattacks, economic attacks, and most obviously information warfare (e.g., creating social discord) will all go to the side with the better AI. One algorithm to rule them all. The US, EU, and China cannot risk losing.
If any one company or country exercises restraint, it must trust the other players to do the same or be willing to accept the consequences if they don’t. Neither corporate boards nor elected officials can take that risk. Minimally, it puts their own jobs at risk. More broadly, the companies and countries themselves may lose out.
And even if somehow the governments all agreed to certain rules, those three governments don’t control the entire cybersphere. North Korea and Iran, both with advanced cyberattack capabilities, would also be independent actors, interested in AI supremacy, even if only partial. A sufficiently rich individual or corporation could likely find a poor nation that would welcome them in exchange for financial support and access to the AI tools. If AI truly can self-improve, it won’t require many people. Hugo Drax, Auric Goldfinger, and Ernst Blofeld suddenly don’t look so far-fetched. (In 2015’s Spectre, it is cybersupremacy that Blofeld leverages.)
Additionally, AI, like anything on the internet, is hard to contain. It is true that the US, EU, and China can exert influence over most of the physical world. A rogue dictator, for example, is at risk of military and/or economic force from one or more of those states. However, online their power is minimal. Once software is released online, it can’t be stopped. The code can be stored on a server in a country where there is limited control. More likely, important code will be copied and stored on multiple servers. Trying to shut it off is like playing whack-a-mole. Nuclear weapons are constrained because enriched uranium takes significant time and money to make. For something that can be replicated by Ctrl + C / Ctrl + V, it will proliferate beyond containment.
Open-source generative AI has typically been only about one year behind the proprietary frontier models. (Note: open-source refers to code being publicly available, while open-weight is often used when LLM weights are publicly available. We’ll use the term open-source to cover both.) Even if all three governments agreed not to pursue further work in AI and to prevent companies from doing so, not everyone would agree. Improvements would continue anonymously, posted into open-source repositories for others to further improve upon and for anyone to use.
You may be thinking, “But they can control the GPUs!” Yes, GPUs are made by a handful of companies that governments can control. But GPUs (graphical processing units) are not the only option. AI can run on CPUs; it’s just not as efficient. It’s like a car running on a gas alternative, it’s not ideal, but it does work. No doubt a determined, well-resourced actor would reduce this level of inefficiency; there’s already been research in this area. Even in the best case, such controls slow down advancement, but don’t restrict current usage. Unless the governments of the world want to try and round up every CPU out there, this is not a limiting factor. (As for data centers, Russian hackers aren’t running their servers on AWS; the dark web offers plenty of options.)
Governments can make frontier development slower, costlier, and less accessible. They can do this effectively within their borders, but with more limited effect externally. It’s not unlike pollution regulation, which governments can directly enforce domestically but can only influence outside their borders. Your country can limit pollution inside its own borders. That helps you in the short term, as you get less direct pollution from a neighboring factory, but, in the long term, if the rest of the world pollutes, you’ll be affected, too. Some people in some countries will be developing ever-better AI. That AI will impact you, directly or indirectly.
There’s no easy solution, and there are many incentives driving us toward suboptimal outcomes. If you believe in “The Better Angels of Our Nature” (as I do), then AI can be a tool to facilitate good. Even if that’s true, it doesn’t mean that it won’t be misused, intentionally or unintentionally, in the short term.
During COVID, governments of the world asked their citizens to wear masks and social distance for the good of society. Many did; quite a few did not. This included people who didn’t believe the risk, didn’t care, were opposed on principle (many with libertarian leanings, a not-uncommon view in Silicon Valley), and some who were just anti-establishment. It also included UK PM Boris Johnson, who simply felt the rules didn’t apply to him and held parties at 10 Downing Street.
How much do you trust your neighbors? How much do you trust leaders of foreign nations? Game theory says that, in a prisoner’s dilemma, we’re led toward defection. Research shows that humans cooperate more often than the math would suggest (see “Cooperation in the Prisoner’s Dilemma: an experimental comparison between pure and mixed strategies,” “Resilient cooperators stabilize long-run cooperation in the finitely repeated Prisoner’s Dilemma,” and “Cooperation through communication: Teams and individuals in finitely repeated Prisoners’ dilemma games”). In the third study, cooperation rates came close to 100%. Unfortunately, when it comes to keeping the genie in the bottle, “close to 100%” likely isn’t sufficient.
This doesn’t mean that nothing can be done, just that we need to be realistic about the limitations of solutions. If anything, it suggests that we may need to employ multiple solutions concurrently.
First, while we may not be able to stop the genie’s escape, we can potentially slow it down and limit some of the damage. Oversight and regulation of the development and deployment of AI, if done right, can potentially limit, or at least delay, some of the harmful effects. The Federal Aviation Administration didn’t prevent all fatalities, but it undoubtedly reduced them through regulations and cooperative learning. Some (but not all) of those dynamics would apply to AI. In parallel, we can strengthen our defenses, such as by using the time to reduce potential zero-day risks, as proposed in The Great Refactor, in which a concerted effort is applied to secure the open-source tools that underpin the modern internet.
Second, we need to better educate society. In The Orville’s Season Three finale, "Future Unknown," Commander Kelly Grayson notes the premise behind their version of Star Trek’s Prime Directive, “Technology and societal ethics have to progress hand in hand, each one supporting the other incrementally. Anything else is begging for disaster.”
I’m a big believer in STEM education because it helps people think analytically and understand how technology and systems generally work. I’m equally convinced that we need history, ethics, philosophy, social studies, and other non-STEM disciplines to help us decide how technology should be used. The United States made a major push to strengthen STEM education over the past several decades. We now need a comparable effort to strengthen the judgment, ethics, and civic understanding required to live with the technologies we are creating. It’s not too late, but the risks increase with each passing day.
It’s critical to learn about corporate culture before you accept a job offer but it can be awkward to raise such questions. Learn what to ask and how to ask it to avoid landing yourself in a bad situation.
Investing just a few hours per year will help you focus and advance in your career.
Groups with a high barrier to entry and high trust are often the most valuable groups to join.
