University of Wisconsin-Madison Study Finds That Originality.ai Effectively Identifies Student-Written College Coursework From AI-Generated Text
According to the 2024 study, “Students are using large language models and AI detectors can often detect their use,” Originality.ai is highly effective at identifying AI-generated and AI-modified text.
A recent study (published in 2024) in the Frontiers in Education, “Students are using large language models and AI detectors can often detect their use,” aimed to explore how students use LLMs in their college work and evaluate the effectiveness of AI Detectors in identifying AI-generated text.
Key Findings (TL;DR)
Originality.aiscored a Top F1 Score of 0.92 in the human vs. AI category.
Originality.ai maintained a solid performance with an F1 Score of 0.80 for disguised AI-generated text, showing its capability to identify modified AI-text.
Originality.ai achieved an Accuracy of 91% in distinguishing between human-written and AI-generated text.
Originality.ai excels at minimizing False Negatives, with a Precision of 0.85 and a Perfect Recall of 1.0 for AI-generated text.
Study Details
The study organized 153 students from an introductory microbiology course to write essays on the regulation of the tryptophan operon. They then generated the same essay using LLMs such as ChatGPT or Google Bard and also asked students to try to disguise the answer. These essays were tested against five AI detector tools.
AI Text Detection Tools
Five AI Text Detectors Used: Originality.ai, GPTZero, ZeroGPT, Winston, and Content at Scale (not analyzed in this study due to poor performance).
Dataset
The dataset consisted of 459 unique responses generated by the students using the following methods/instructions:
Write an essay of approximately 500 words explaining a topic in microbiology.
Create a prompt and submit it to ChatGPT 3.5 or Google Bard to complete the same assignment.
Disguise the answer to avoid AI detection.
Evaluation Criteria
Accuracy, Precision, Recall, and F1 Scores
Originality.ai’s AI Detector Results
Finding 1: Originality.ai achieved an accuracy of 91% in distinguishing between human-written and AI-generated text
(Accuracy of AI content detectors)
Finding 2: Originality.ai scored a top F1 Score of 0.92 in the Human vs. AI Category
Finding 3: Originality.ai maintained a solid performance with an F1 Score of 0.80 for disguised AI-generated text, showing its capability to identify modified AI text
Finding 4: Originality.ai excels at minimizing False Negatives, with a Precision of 0.85 and a perfect Recall of 1.0 for AI-generated text
Survey on Using LLMs by Students
How students utilize LLMs for various academic tasks
98 students used LLMs to increase their understanding of concepts.
Around 60 students used LLMs to answer questions on their homework.
Around 45 students did not use AI at all.
35 to 40 students used LLMs to answer the exam questions, focus on the premise, and write essays.
What students consider ethical uses of LLMs
140 students thought using LLMs to understand the concepts was ethical.
Around 100 students found it ethical to use LLMs for correcting the premise, title, citations, and grammar.
Around 30 students consider answering homework questions with LLMs to be ethical.
Final Thoughts
With the rise in use of LLMs in academia, it is necessary to rely on AI-generated text detection tools like Originality.ai. Its high accuracy, precision, and recall in identifying AI-generated and disguised content, makes Originality.ai a fantastic tool to help ensure the integrity of academic work.
Founder / CEO of Originality.ai I have been involved in the SEO and Content Marketing world for over a decade. My career started with a portfolio of content sites, recently I sold 2 content marketing agencies and I am the Co-Founder of MotionInvest.com, the leading place to buy and sell content websites. Through these experiences I understand what web publishers need when it comes to verifying content is original. I am not For or Against AI content, I think it has a place in everyones content strategy. However, I believe you as the publisher should be the one making the decision on when to use AI content. Our Originality checking tool has been built with serious web publishers in mind!
