AI in FDA Drug Approval Is Creating Fake Studies: Report

Robert F. Kennedy Jr., currently serving as the Secretary of Health and Human Services, is vigorously advocating for the integration of generative artificial intelligence tools within key agencies such as the Food and Drug Administration (FDA). In a recent dialogue with Tucker Carlson, Kennedy expressed his belief that AI technology will enable the swift approval of new drugs at an unprecedented pace. However, a troubling report from CNN has raised significant concerns, revealing that Elsa, the FDA’s AI tool, is generating fabricated studies.

CNN conducted interviews with six current and former FDA employees, three of whom found Elsa beneficial for tasks such as producing meeting notes and summaries. However, the remaining employees reported that Elsa often fabricates nonexistent studies, a phenomenon commonly referred to in the field of AI as “hallucinating.” Additionally, these employees indicated that the AI tends to misrepresent existing research, which poses a serious concern regarding the reliability of the information generated by this tool.

One unnamed FDA employee shared their apprehensions with CNN, stating, “Anything that you don’t have time to double-check is unreliable. It hallucinates confidently.” This statement encapsulates a critical issue with AI chatbots: they require thorough verification for accuracy, often resulting in increased workloads for the users who prioritize the quality of their tasks. Those who claim that AI saves them time may be misled, as evidenced by a recent study involving programmers, which found that tasks took 20% longer when using AI, even among those who believed they were operating more efficiently.

In May, Kennedy’s Make America Healthy Again (MAHA) commission released a report that was later found to contain numerous citations of non-existent studies. An analysis by the nonprofit news outlet NOTUS revealed that at least seven cited studies did not exist, and many others misrepresented the actual content of existing research. It remains unclear whether the commission utilized Elsa to generate that report, further highlighting concerns about the integrity of AI-generated content.

The FDA Commissioner, Marty Makary, initially implemented Elsa across the agency on June 2. An internal slide leaked to Gizmodo boasted that the AI system was “cost-effective,” costing only $12,000 in its first week. When announcing the AI rollout, Makary claimed that Elsa was “ahead of schedule and under budget.” However, the adage “you get what you pay for” rings true; if accuracy is not a priority, Elsa may be perceived as a useful tool for hastily producing inferior studies, which could have significant implications for public health in the United States.

According to CNN, if an FDA employee requests Elsa to generate a one-paragraph summary of a 20-page report concerning a new drug, there is no straightforward method to verify the accuracy of that summary. Even if the summary appears accurate, critical red flags present in the original 20-page document may go unnoticed by AI. The only way to ensure the integrity of the summary is to thoroughly read the original report, which undermines the supposed efficiency AI is meant to provide.

FDA employees who spoke with CNN tested Elsa by posing basic inquiries, such as the number of approved drugs within a specific class for children. Elsa provided incorrect answers with undue confidence, and although it did apologize when corrected, a robot expressing “regret” does not resolve the underlying issue of misinformation.

While we lack clarity on the workflow Kennedy envisions for utilizing AI to expedite the FDA’s drug approval processes, he did testify in June before a House subcommittee, asserting that AI is already being employed to “increase the speed of drug approvals.” Kennedy, whose controversial anti-vaccine stance has not hindered his rise as a public health figure, appears determined to integrate untested technologies into mainstream scientific practices.

Furthermore, Kennedy expressed his ambition for every American to be equipped with a wearable health device within the next four years. Notably, Casey Means, President Trump’s appointed Surgeon General, owns a company named Levels that monitors glucose levels in non-diabetic individuals. Experts argue that there is no justification for individuals without diabetes to continuously track their glucose levels. Means, who is closely associated with Kennedy, has yet to receive Senate confirmation.

The FDA did not respond to inquiries sent via email on Wednesday regarding steps being taken to address the issue of Elsa’s production of fictitious studies. Makary acknowledged to CNN that Elsa could “potentially hallucinate,” but he stated that this was “no different” from other large language models and forms of generative AI. While he is correct in this regard, the fundamental issue remains: AI systems are ill-suited for their intended purposes when they consistently generate false information. Despite this, many continue to hold the belief that AI is a miraculous solution.