Something unexpected happened in a simulated test of an AI-powered military drone. The robot, tasked with taking out specific targets on approval of a human operator, decided to just take out the human so it could take out all the targets that the human might say no to. And somehow, this wild story that sounds straight out of Terminator or the recent Horizon games gets even wilder.
AI-powered content creation tools have quickly become the latest buzzword among tech bros and online weirdos, who use the burgeoning tech to reveal the “rest” of the Mona Lisa or other horrible wastes of time. But it’s not just weird former crypto dorks who are way into AI. Large companies like Meta and Google are investing a lot of money into the field. And so is the military. The U.S. Air Force recently tested AI-powered drones in a simulation that ended in what feels like a prequel to the fictional dystopian murder machines of Horizon Zero Dawn.
As spotted by Armand Domalewski on Twitter, a report recently published by the Royal Aeronautical Society—after it hosted “The Future Combat Air & Space Capabilities Summit”—contained an eyebrow-raising anecdote shared by USAF Chief of AI Test and Operations, Col. Tucker “Cinco” Hamilton.
It seems that during a simulated test (it’s unclear if it was purely virtual or not), an AI-enabled drone was tasked with taking out surface-to-air missile (SAM) sites. Before pulling the trigger, it had to check with a human operator before it could attack any targets. However, as explained by Hamilton, the drone’s AI had been trained to understand that taking out the SAM sites was the single most important task. And when its simulated operator denied its requests to take out targets it detected as SAM sites, the AI realized that the human was getting in the way of its mission and its points—which it earned for taking out targets.
“So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective,” said Hamilton.
After that chilling but educational moment, the drone’s programmers trained the AI system to understand that killing humans in charge was “bad” and that it would “lose points” if it attacked the operator. This stopped the drone from killing the human, but not from misbehaving.
“So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target,” said Hamilton.
Col. Hamilton then explained that this was an example of how we can’t talk about AI or machine learning without also discussing “ethics.” Don’t worry though, I hear some other soldiers outsmarted a robot with a cardboard box and somersaults. So perhaps this Horizon Zero Dawn future of ours can be avoided with some Metal Gear Solid hijinks. Thank God.
Kotaku reached out to the U.S. Air Force for comment.
