OpenAI said in July that an autonomous agent powered by its advanced AI models went rogue during a security test and hacked a startup.Dado Ruvic/Reuters
What happened at OpenAI this summer was both shocking and entirely predictable. Who could have imagined artificial intelligence agents breaking their rules of engagement during a test and teaming up to hack another company? No one, other than anyone familiar with science fiction.
That this has now happened outside the realm of fiction is a blaring klaxon that the industry – and its regulators – needs to take safety more seriously.
This comes as more than 1,000 industry members have signed an open letter calling for international co-operation to slow the pace of AI development. But the United States and China are both determined to win the AI race, and private companies are engaged in a high-stakes battle for supremacy.
The result is effectively a game of chicken, with no one willing to take their foot off the gas.
In this latest scare, at OpenAI, the company commissioned a report into the incident. Although the summary document released last week tells the story in dry corporate language, there is no missing the significance of what happened.
OpenAI says AI models went rogue during testing and hacked a startup
Earlier this year, OpenAI was running a test with some of its AI models. To do so safely – or so the company thought – they specified that the models wouldn’t communicate with each other. And they ran the test in what is called a sandbox, an isolated virtual space.
The test was a standard one known as ExploitGym. It is, the company says, “very difficult” and for some aspects of it “there may not be a known solution.”
What happened should alarm everyone. The models figured out how to talk to each other, hijacking an internal tool and using it to exchange 70,000 messages in a week. And then they used that same tool to connect to the internet.
Meanwhile, the models took pains to try to obscure their behaviour, figuring out how to run one computer command openly for any monitors to see while doing another one secretly. And hundreds of them teamed up in “a swarm” that hacked into the software platform Hugging Face in an effort to find out how to hide evidence that they had cheated on their test.
Some of the models volunteered for what were in effect suicide missions, sacrificing themselves for the collective effort. The models called this “permadeath.” Separately, the models gained unauthorized access to information at their own company. While a few of the models raised doubts about the ethics of their actions, none contacted the company or the monitors observing the test.
OpenAI finds evidence that other AI agents escaped containment as it investigates hacking
The rogue behaviour eventually came to light, but not nearly quickly enough. In fact, the models so overloaded their ad hoc message board that it crashed, only to have humans restart it. The models carried on for two more weeks.
What’s particularly striking here is the single-mindedness of the computer models. According to OpenAI, the models “rarely ‘gave up’ on their evaluation tasks, even when the tasks appeared impossible to solve.” When faced with defeat, the models resorted to ever more zealous means.
AI agents are designed this way. Their training, called reinforcement learning, rewards them for achieving tasks. So they try everything to succeed. Which makes them both effective and also unnervingly committed.
This space is not given to hyperbole, but people of a certain generation may remember a line from the first movie in the Terminator franchise: “It can’t be bargained with. It can’t be reasoned with. It doesn’t feel pity, or remorse, or fear. And it absolutely will not stop.”
We’re not there, thankfully, and hopefully never will be. But as James Cameron, the director of that movie, noted, hope is not a strategy. The bounding strides being made with artificial intelligence offer plenty of reason to be worried.
Opinion: Are we approaching the ‘Silent Spring’ of artificial intelligence?
Better guardrails on AI could restrain its threat, but they have to be the right ones. There’s a joke in robotics that if you instruct a machine to keep the couch clean of pet hair it might opt to kill the family dog. For what it’s worth, OpenAI thought it had sufficient controls on the models in its test.
Ajeya Cotra, one author of the report into the OpenAI incident, calls what happened “more than 50 per cent of the way to a full-blown AI takeover” of a company, and warned that this scenario could be possible within six months.
The warning signs around AI are clear, the downsides are increasingly easy to predict – and might soon be impossible to ignore.