Signs That AI Wrote a Paper: Insights from a Professor

Mark Massaro has dedicated many years to teaching English Composition at Florida Southwestern State College. However, in 2023, his role became significantly more challenging due to the emergence of AI applications like ChatGPT, which have led to a surge in automated cheating within higher education across the United States. Students have increasingly been using AI technology to generate essays, and this troubling trend seems to be intensifying over time. Massaro reported that out of 25 students in his classes, it is not uncommon for up to five to submit papers that clearly exhibit the characteristics of AI-generated content.

Following the launch of ChatGPT, it quickly became evident that students were exploiting its capabilities to cheat. Initially, Massaro would scan numerous papers daily using various AI detection tools. If a significant number of these tools indicated that the writing was AI-generated, he would trust the results. However, the situation has evolved, as the school has prohibited the uploading of students’ work to such applications due to potential privacy violations. Moreover, the accuracy of AI detectors has proven to be quite unreliable, making it even more difficult for educators like Massaro to navigate this new landscape.

Now, Massaro relies on his own judgment to determine whether a paper has been illegitimately created using AI. To assist in this process, he has developed a checklist of common indicators that suggest a paper may have been generated by AI. Whenever a student submits a paper, he refers to this list to identify any potential red flags. He shared several of these warning signs with Gizmodo, shedding light on the challenging environment that AI technology has imposed on educators in America.

Recognizing Excessive Em-Dashes as a Sign of AI Writing

One significant indicator that a paper may have been generated by AI is the presence of an excessive number of em dashes. For some reason, applications like ChatGPT tend to incorporate a surplus of these punctuation marks in their writing. Massaro has devised a practical test: if he suspects a student has used AI to generate their essay, he will call them to his desk and ask them to demonstrate how to create an em dash on the computer. Often, students struggle with this simple task, which raises red flags for him. “I’ll say, your paper is filled with em dashes; just show me how you did it,” he observed. If the student falters, it becomes clear that something is amiss.

Identifying the Lack of Paragraph Indentation in AI-Generated Texts

Massaro points out that, contrary to the traditional formatting guidelines for college essays, it is common for texts copied and pasted from a chatbot to lack proper paragraph indentations. The output produced by ChatGPT and similar tools often does not require these indentations. If a student submits a paper characterized by large, unbroken blocks of text, there is a strong likelihood that it has been generated by a robot rather than written by a human. Recognizing these formatting discrepancies is crucial for educators as they strive to uphold academic integrity in their classrooms.

Detecting Flawless Grammar but Superficial Content in Student Papers

Another significant warning sign that may indicate a paper has been algorithmically generated is the submission of a flawlessly written essay that lacks depth and originality. While some students may excel in writing yet struggle to express unique thoughts, AI-generated writing typically carries distinctive markers that become evident over time. Massaro describes this AI writing style as having “uniform sentence and paragraph length,” where each paragraph tends to be of similar size, giving the writing a rhythmic, mechanical quality. Additionally, it often adopts a “hyper-formal tone,” characterized by an overly polished academic voice that is not representative of typical student work.

Spotting the Absence of Drafting History in Student Assignments

Another red flag to watch for is the complete lack of any involvement in the in-class drafting process. As part of his composition courses, Massaro conducts peer reviews and conferences to foster a collaborative writing environment. A student might also assert that they consulted with the school’s Writing Center to craft their essay, but in reality, they may not have done so at all. When students submit these papers, they often contain “unusual vocabulary for the student,” demonstrating “sudden jumps in sophistication” that are not consistent with their prior work or class discussions. This inconsistency can further indicate the use of AI tools in essay creation.

Understanding the Impersonal Nature of AI Writing

Massaro occasionally assigns personal reflection papers to students, encouraging them to draw from their own experiences. A student who genuinely writes about their time working at a local 7-Eleven, for instance, will weave in various personal anecdotes that an AI would struggle to fabricate. Conversely, if a paper is generated by AI, it may discuss abstract concepts, such as the importance of “friendship” in a broad and impersonal manner, lacking any genuine personal touch. This stark difference in tone and content can help educators identify the authenticity of student work.

Identifying Leftover AI Prompt Inputs in Student Submissions

In some surprising cases, students have been remarkably careless with their cheating, leaving behind prompt inputs from their interactions with a chatbot in the final essay they submitted, according to Massaro. Such oversights can serve as glaring indicators that the work is not genuinely theirs, further complicating the task of maintaining academic integrity.

Recognizing Fabricated Citations and Information in AI-Written Essays

It is widely known that AI has a tendency to “hallucinate” information, which can manifest in various ways within a student’s paper. These AI-generated references can include blatantly incorrect or entirely fabricated information, as well as fake citations. This phenomenon often results in hours of additional work for Massaro, who must verify whether an obscure fact included in an essay is legitimate. His investigation may involve scrutinizing real academic journals and books referenced in the student’s work, only to discover that the citations in the bibliography are fabricated.

Addressing the issue of cheating with a student can be an uncomfortable and challenging experience, Massaro noted. However, he often recognizes the signs of AI-generated work right away. In many cases, a guilty student may concede that they used AI, or they may simply not respond to Massaro’s emails (as many of his students are remote). Conversely, innocent students frequently protest their integrity, and Massaro tends to believe their claims. The most disheartening aspect of the rise of AI writing is that it deprives students of the opportunity to develop their own intellectual and creative identities, he lamented. “This is the time when they’re supposed to be finding their voice,” he said. Instead, AI is doing the speaking (and the thinking) for them.