Listen to the article 2 min

This audio is auto-generated. Please let us know if you have feedback.

Dive Brief:

• Generative artificial intelligence tools are making their way into classroom learning as teachers put them to use for planning lessons and creating assignment prompts. 

• However, experts warn that AI hallucinations — which occur when AI presents an incorrect or misleading response as fact — can crop up when tasking these tools with writing a biography or checking the results of a math problem. Researchers from Stanford and Yale universities found, for instance, that AI legal research tools from Lexis Nexis and Thompson Reuters produced hallucinations between 17% and 33% of the time.

• Daniel Ho, one of the paper’s authors and a professor of law, political science and computer science at Stanford University, said that there needs to be “a lot of evaluation to see where AI systems can be reliable and helpful.”

Dive Insight:

A great deal of attention has been given to students using generative AI in their work, ranging from employing it as a research tool to using it to write entire essays — the latter of which has also sparked debate over academic integrity.

Less attention has been given to the challenges educators may face with the results they get when using it to help develop lessons and learning materials.

Fact-checking the results produced by generative AI tools, such as answers given to queries by ChatGPT — is a best practice for educators and students alike. However, why the hallucinations occur in the first place is a question Ho and his colleagues considered in their research released earlier this year.

Sycophancy can be one reason, the authors noted, as a “large language model” AI tool like ChatGPT may agree at the outset with a user “even when the user is mistaken,” the researchers wrote.

“One of the more challenging forms of hallucination emerges from the sycophantic nature of language models. They are trained to be obsequious to users,” said Ho, who also serves as a senior fellow at the Stanford Institute for Human-Centered Artificial Intelligence.

“When users ask questions with mistaken premises, for instance, AI may uniquely struggle in correcting those premises,” said Ho.