Dive Brief:
- Nearly two months after releasing an artificial intelligence writing detection feature, plagiarism detection service Turnitin has reported a “higher incidence of false positives” — or incorrectly identifying fully human-written text as AI-generated text — when less than 20% of AI writing is detected in a document.
- However, Annie Chechitelli, Turnitin’s chief product officer, did not disclose the exact rate of discovered false positives in the company’s post. As of May 14, 38.5 million submissions have gone through the tool, with 9.6% of those documents reporting over 20% of AI writing and 3.5% over 80% of AI writing, Chechitelli wrote.
- To address concerns over false positives, Turnitin will display an asterisk when the detector spots under 20% of AI writing. The move will let teachers know the score is less reliable in those scenarios, according to Chechitelli.
Dive Insight:
Upon launching the AI detection tool, Turnitin expressed a 98% confidence rate in it finding AI writing. The company’s tool emerged about five months after ChatGPT, the explosively popular generative AI tool, debuted in November.
Though it’s still unclear how much that confidence rate has changed since the tool’s release, Chechitelli said Turnitin “discovered real-world use is yielding different results from our lab.”
Turnitin is also increasing the minimum word count — from 150 to 300 words — to be evaluated by its AI detection tool. “Results show that our accuracy increases with a little more text, and our goal is to focus on long-form writing,” Chechitelli wrote. She added that the word count minimum may change later upon further evaluation.
To further cut down on false positives, Chechitelli said Turnitin is changing how it aggregates sentences at the beginning or end of a document, because that’s where higher rates of false positives appear.
Concerns over detecting false positives have been more publicized in higher education, but remain worrisome for educators at all levels. With that in mind, Turnitin released new guidance for educators on how to handle false positive scenarios, including how to approach conversations with students.
At Illinois’ Hinsdale Township High School District 86, English teachers have already found several students using generative AI in their writing assignments — as spotted by the Turnitin tool, said Keith Bockwoldt, the district’s chief information officer. As of May, Bockwoldt said, the district had not run into any issues with false positives, however.
At this time of year, teachers know their students’ writing styles and can better spot if AI is used, Bockwoldt said. But in the fall, teachers won’t have that reference point when they begin teaching a new group of students, he said.
In addition to having teachers use Turnitin’s AI detector, Bockwoldt said the district has encouraged instruction about appropriate use of AI along with lessons on academic integrity and citing sources.
