Malicious Documents Expose LLMs to Backdoor Vulnerabilities

Key Points

• Findings: The study highlights that only a small number of malicious documents can poison a large language model (LLM).
• Type of Attack: The study focuses on data-poisoning attacks that compromise the integrity of LLMs.
• Collaboration: Anthropic worked with the UK AI Security Institute and the Alan Turing Institute for this research.
• Impact: The results suggest that data-poisoning attacks might be more feasible than previously thought.

Artificial intelligence companies have been working at breakneck speeds to develop the best and most powerful tools, but that rapid development hasn’t always been coupled with clear understandings of AI’s limitations or weaknesses. Today, Anthropic released a report on how attackers can influence the development of a large language model.

What is Data Poisoning in LLMs?

Data poisoning is a type of attack where a large language model is pretrained on harmful content designed to teach it undesirable behaviors. This study reveals that even a minimal quantity of malicious documents can effectively compromise an LLM’s integrity.

The core takeaway is that a small number of malicious inputs can lead to significant vulnerabilities in LLMs.

The study centered on a type of attack called poisoning, where an LLM is pretrained on malicious content intended to make it learn dangerous or unwanted behaviors. The key finding from this study is that a bad actor doesn’t need to control a percentage of the pretraining materials to get the LLM to be poisoned. Instead, the researchers found that a small and fairly constant number of malicious documents can poison an LLM, regardless of the size of the model or its training materials. The study was able to successfully backdoor LLMs based on using only 250 malicious documents in the pretraining data set, a much smaller number than expected for models ranging from 600 million to 13 billion parameters.

Why is This Research Important?

This research is crucial as it sheds light on the practicality of data-poisoning attacks, potentially reshaping how developers approach LLM security. The findings indicate a pressing need for enhanced defenses against such vulnerabilities.

Anthropic’s research serves as a call to action for further exploration of data poisoning and protective measures.

“We’re sharing these findings to show that data-poisoning attacks might be more practical than believed, and to encourage further research on data poisoning and potential defenses against it,” the company said. Anthropic collaborated with the UK AI Security Institute and the Alan Turing Institute on the research.