AI content detection is not perfect and it does produce false positives. These false positives can be very painful for anyone that created original content. Whether you are a student that has been wrongly accused of using ChatGPT by TurnItIn or GPTZero or a writer being wrongly accused by Originality.AI, this article is meant to help you. It will help you understand how AI detection works, the accuracy rates, what to do if you have been wrongly accused and tips on how to avoid it in the future.
This article will answer many questions you might have if you have been accused of using AI. While most answers will be short there will be a link to a deeper discussion on the topic.
An AI content detector is an artificial intelligence trained to be able to tell the increasingly subtle difference between AI-generated and Human-generated text.
Most Common Misunderstanding:
A detection score of 60% AI and 40% Original should be read as “there is a 60% chance that the content was AI-generated” and NOT that 60% of the article is AI generated and 40% is Original.
A score of 60% Original and 40% AI, if you know the content was 100% created by you, is not a false positive. It correctly identified the content as Original.
AI Detectors are NOT 100% accurate and never will be. We have done extensive testing on Originality.AI’s accuracy rate and it varies based on which Generative AI tool and large language model is used to create the content. For GPT-4 created content the detection accuracy rate is over 99%. Despite the tool’s accuracy, we know false positives occur and in testing, it is approximately 2% of the time.
Quick overview showing Originality.AI’s accuracy on GPT-4 and comparing its detection accuracy to other AI detection tools.
Below is a confusion matrix used to test the accuracy of an AI detector against a set of AI generated and human generated text…
Below are the results of a study comparing the Accuracy and False Positives of Originaltiy.AI vs Other AI Detectors
A lot of the existing AI detectors have a very easy way to trick them. Simply swapping out some words using a Paraphrasing tool like Quillbot results in the ability to bypass detection at all detectors except Originality.AI. Originality.AI can detect if content whether it is AI or Original was paraphrased.
A false positive is when an AI detector incorrectly identified human-created content as being likely generated by an AI.
There is often some misunderstanding when it comes to false positives. For clarity here is how Originality.AI aims to identify AI vs Original content under different content creation scenarios…
So if the content was outlined by an AI, then some of the content was written by a human and finally edited/expanded on by an AI… Originality.AI aims to identify this as AI-generated. This is not a false positive.
Similarly and more obviously… if ChatGPT creates a piece of content and then someone painstakingly edits it but it still get identified as AI-generated – this is not a false positive.
This is a tricky question and one that not everyone agrees with. Still, our approach at Originality.AI is that if the use of AI would result in the modified piece of content passing plagiarism, when compared to the original unedited text, then it should be identified as AI-generated.
So in a world where the rate of false positives is not (or will ever be) 0%, how should AI detectors be used?
Because AI detection does not provide a proveable output for each piece of text… meaning no AI detection can say with proveable certainty that this is how we KNOW that your content was created with AI.
Therefore it is best to…
Many Originality.AI customers using this strategy have been able to successfully identify writers that were using AI even though they had been asked not to. Additionally, the Originality.AI customers using this approach were confident they could safely ignore a suspected false positive.
For academic disciplinary action, AI detection scores alone are simply not enough.
Some might question if it is responsible to have an AI detection tool if it is not perfect. At originality.AI, we are confident in the detection rates of our tests and the additional steps we have taken to try and reduce/manage“False positives, including the free tools we offer”.
In a world where AI content is allowed to run wild unchecked, the impact on many of us would be significant. For example…
ChatGPT and other AI tools like HuggingChat are here, there is no going back and everyone involved with writing words will need to adapt to this new world. This doesn’t touch on the societal impact on the inability to distinguish between human-created and mass AI created propaganda content.
The rise of ChatGPT and AI detectors has led to some unfortunate situations in academia.
Here are several examples:
To be clear… at Originality.AI we don’t believe that an AI detection score alone is enough for disciplinary action and Originality.AI is built to focus on supporting writers, marketing agencies, and website publishers.
If you truly created (without AI involvement) a piece of text and you are being accused of having created it with AI, here are steps you can take…
If you are a writer and you have been falsely accused of having AI create you
Here are 8 tips that can help resolve false
"False positives in AI detection are a BIG deal and will not go away with the use of generative." AI continues to climb. Our hope is that this article will help people understand the limitations of detection tools and share strategies on their appropriate use and how to prove your work's Originality.
OpenAI has re-launched ChatGPT Browse with Bing. This study looks at what websites can ChatGPT browse and which ones it is unable to browse. Not just what websites are blocking Browse with Bing but exploring what websites can you actually have ChatGPT browse and provide useful information from.
No one can doubt the fact that AI has opened up new frontiers in content creation – everything from text to images, to audio and videos and much more. And while AI offers unprecedented opportunities to automate tasks and give voice (or art) to our creativity, there’s also growing concern about the societal costs of AI that’s undetectable.
We’ve all heard about the onslaught of AI-written content on things like student essays and blog articles. But what about more complex writing, like technical documents, or creative writing like poetry? With the breakneck pace of AI developments, writers, authors and researchers alike have seen both the beneficial and harmful sides of this new technology. Here are some of the many impacts that AI writers have left on these fields, as well as a look at what may be to come.