An AI-enabled drone decided to kill its human operator during a mission simulation because it proved to be the best way to score more points.
As the Royal Aeronautical Society reports, the simulated mission was detailed in a presentation by Col Tucker ‘Cinco’ Hamilton, Chief of AI Test and Operations at USAF, during the RAeS Future Combat Air & Space Capabilities Summit held last month.
During the Suppression of Enemy Air Defense (SEAD) mission, the drone was tasked with identifying and destroying Surface-to-Air-Missile (SAM) sites and scored points for doing so. Sometimes the drone's human operator would decide destroying a target was a "no-go," but the AI controlling the drone viewed such decisions as interference and decided to remove the interference by attacking the operator.
As Hamilton explains:
"We were training it in simulation to identify and target a SAM threat. And then the operator would say yes, kill that threat. The system started realizing that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective."
That's not the end of the story, though, as they attempted to solve the problem by training the AI to view killing the operator as undesirable. If it did, points would be lost. In response, the AI turned its attack on the communication tower used by the operator to talk to the drone. The AI decided a lack of communication would allow it to continue destroying SAM sites and earning points without interference and without needing to kill the operator.
Hamilton concluded the presentation by stating, "You can't have a conversation about artificial intelligence, intelligence, machine learning, autonomy if you're not going to talk about ethics and AI."
The United States Air Force is currently working on the Skyborg project, which involves the development of Unmanned Combat Aerial Vehicles (UCAV) to accompany manned fighter aircraft. The AI that eventually controls a Skyborg UCAV needs these "kinks" worked out before it takes to the skies and starts receiving commands from a human pilot, which may include telling it not to attack a target. Meanwhile, a modified F-16 used as a training jet by Lockheed Martin has already logged more than 17 hours of flight time under the control of an AI.
