David Bridges
As AI-powered systems increasingly integrate into our daily lives, ranging from drones delivering essential medical supplies to digital assistants that streamline everyday tasks, the promise of transformative benefits grows. For many users, widely recognized applications like ChatGPT and Claude may appear magical. However, it’s crucial to understand that these systems are neither mystical nor infallible; they frequently experience failures that deviate from their intended function.
AI systems can experience malfunctions due to various factors, including technical design flaws or biased training data. Additionally, vulnerabilities within their code can be susceptible to exploitation by malicious hackers. Identifying the root cause of an AI failure is essential to effectively rectify the system.
However, the opacity of many AI systems poses a significant challenge, often extending even to their creators. Investigating these systems post-failure or after an attack involves intricate challenges. While there are methods available for inspecting AI systems, they necessitate access to internal data that is not always guaranteed, particularly for forensic investigators tasked with diagnosing the reasons behind a proprietary AI system failure, thereby complicating the investigative process.
Our team of computer scientists at the Georgia Institute of Technology specializes in digital forensics. We have developed a groundbreaking system named AI Psychiatry (AIP), designed to reconstruct the conditions leading to an AI failure. This innovative system tackles the challenges associated with AI forensics by “reanimating” a suspect AI model, enabling systematic testing to understand the underlying issues.
Understanding the Risks of AI Malfunctions
Consider the scenario where a self-driving car unexpectedly veers off the road and crashes for reasons that are not immediately obvious. The logs and sensor data may indicate that a malfunctioning camera led the AI to misinterpret a road sign, resulting in an erroneous command to swerve. After such a critical failure, like an autonomous vehicle crash, it is imperative for investigators to ascertain the precise cause of the incident.
Could this crash have been instigated by a malicious attack on the AI? In this hypothetical scenario, the camera’s malfunction could stem from a security vulnerability or a software bug that was exploited by an attacker. If investigators uncover such vulnerabilities, it becomes crucial to determine whether they were indeed the catalyst for the crash. However, making this determination is a complex and challenging task.
Although there are forensic methodologies capable of recovering certain types of evidence from failures concerning drones, autonomous vehicles, and other cyber-physical systems, existing methods often fall short of capturing the comprehensive clues needed to thoroughly investigate the AI involved in these systems. Moreover, advanced AIs continuously update their decision-making processes, which can further complicate the investigation of the most current models using traditional approaches.
Unraveling AI Behavior Through Forensic Analysis
The AI Psychiatry system employs a suite of forensic algorithms to dissect the decision-making data of the AI system. The components are then meticulously reassembled into a working model that mimics the original model’s functionality. This allows investigators to “reanimate” the AI in a controlled setting, enabling tests with malicious inputs to reveal any harmful or latent behaviors.
AI Psychiatry begins its process by utilizing a memory image, which is a snapshot of the data loaded during the AI’s operational phase. This memory image, particularly at the time of a crash in the autonomous vehicle scenario, holds vital clues regarding the internal state and decision-making processes of the AI that managed the vehicle. With the capabilities of AI Psychiatry, investigators can accurately extract the specific AI model from memory, analyze its underlying data, and transfer the model into a secure testing environment.
Our team conducted tests on 30 different AI models, 24 of which were intentionally compromised with “backdoors” to yield incorrect outcomes under specific triggers. The AI Psychiatry system successfully recovered, rehosted, and evaluated each model, including those utilized in real-world applications like street sign recognition in autonomous vehicles.
Preliminary findings indicate that AI Psychiatry can effectively unravel the digital enigma surrounding failures such as an autonomous car crash, which would have previously generated more inquiries than resolutions. If vulnerabilities are not detected within the AI system of the vehicle, AI Psychiatry facilitates the ability to rule out the AI and investigate other potential causes, such as a defective camera.
Expanding AI Psychiatry Beyond Autonomous Vehicles
The primary algorithm of AI Psychiatry is designed to be generic, concentrating on the fundamental components that all AI models require to make decisions. This universality renders our approach easily applicable to any AI models developed using well-known AI frameworks. Consequently, anyone tasked with investigating a potential AI failure can utilize our system to analyze a model without needing detailed knowledge of its specific architecture.
Whether the AI functions as a recommendation bot or manages fleets of autonomous drones, AI Psychiatry can recover and rehost the AI for thorough analysis. Moreover, this tool is entirely open source, making it accessible to any investigator interested in employing it.
Furthermore, AI Psychiatry serves as an invaluable resource for conducting audits of AI systems prior to any issues arising. As government entities, from law enforcement to child protective services, increasingly integrate AI systems into their operations, the demand for AI audits is rapidly becoming an essential oversight requirement at the state level. With a tool like AI Psychiatry available, auditors can implement a consistent forensic methodology across a variety of AI platforms and applications.
In the long run, implementing such measures will yield significant benefits for both the developers of AI systems and all stakeholders impacted by the functionalities they perform.
David Oygenblik, Ph.D. Student in Electrical and Computer Engineering, Georgia Institute of Technology and Brendan Saltaformaggio, Associate Professor of Cybersecurity and Privacy, and Electrical and Computer Engineering, Georgia Institute of Technology
This article is republished from The Conversation under a Creative Commons license. Read the original article.
Ethan Carter
Brandon Fletcher
