David Bridges
OpenAI has taken decisive action by banning the accounts of a group of Chinese users who attempted to exploit ChatGPT for debugging and editing code related to an AI social media surveillance tool. This initiative, which OpenAI refers to as Peer Review, involved the users prompting ChatGPT to create persuasive sales pitches for a program that documentation indicates was intended to monitor anti-Chinese sentiment across platforms such as X, Facebook, YouTube, and Instagram. The primary focus of this operation was to identify and track calls for protests against human rights violations occurring within China, with the goal of relaying these insights to the government.
OpenAI outlined that this network comprised ChatGPT accounts that operated according to a schedule aligned with mainland Chinese business hours, engaged in prompting the models in Chinese, and utilized the tools with a volume and diversity typical of human interaction rather than automated processes. “The operators leveraged our models to validate claims that their gathered insights were forwarded to Chinese embassies abroad, as well as to intelligence agents monitoring protests in various countries, including the United States, Germany, and the United Kingdom,” OpenAI stated.
Ben Nimmo, a principal investigator at OpenAI, emphasized that this incident marks the first discovery of an AI tool used in such a manner. “Threat actors occasionally provide us with insights into their operations on other parts of the internet, which we can observe through their interactions with our AI models,” Nimmo explained. This revelation raises significant concerns about the potential misuse of AI technology for surveillance and control.
The code for the surveillance tool appears to be largely derived from an open-source version developed by Meta. Furthermore, the group reportedly utilized ChatGPT to produce a year-end performance review, which included claims of drafting phishing emails on behalf of clients based in China. This highlights the alarming versatility of AI applications, which can be repurposed for malicious activities.
OpenAI remarked, “Evaluating the repercussions of this activity necessitates contributions from numerous stakeholders, including operators of any open-source models, who can provide insight into this troubling behavior.” The company’s statement underscores the importance of collaboration to understand and mitigate the risks associated with the misuse of AI technologies in surveillance operations.
In a related matter, OpenAI announced the recent banning of an account that employed ChatGPT to generate social media posts critical of a Chinese political scientist and dissident currently living in exile in the United States. This same group also utilized the chatbot to craft articles in Spanish that criticized the United States. These articles were disseminated by mainstream news outlets in Latin America and were often attributed to specific individuals or a Chinese company.
Daniel Mercer
Ethan Carter
