AI Content Detector False Positives – Accused Of Using Chat GPT Or Other AI?

AI content detection is not perfect and it does produce false positives. These false positives can be very painful for anyone that created original content. Whether you are a student that has been wrongly accused of using ChatGPT by TurnItIn or GPTZero or a writer being wrongly accused by Originality.AI, this article is meant to help you. It will help you understand how AI detection works, the accuracy rates, what to do if you have been wrongly accused and tips on how to avoid it in the future.

This article will answer many questions you might have if you have been accused of using AI. While most answers will be short there will be a link to a deeper discussion on the topic.

What is AI Content Detection?

An AI content detector is an artificial intelligence trained to be able to tell the increasingly subtle difference between AI-generated and Human-generated text.

• How does our AI Detector Work

Most Common Misunderstanding:

A detection score of 60% AI and 40% Original should be read as “there is a 60% chance that the content was AI-generated” and NOT that 60% of the article is AI generated and 40% is Original.

• What AI Detection Scores Mean

A score of 60% Original and 40% AI, if you know the content was 100% created by you, is not a false positive. It correctly identified the content as Original.

How Accurate are AI Detectors?

AI Detectors are NOT 100% accurate and never will be. We have done extensive testing on Originality.AI’s accuracy rate and it varies based on which Generative AI tool and large language model is used to create the content. For GPT-4 created content the detection accuracy rate is over 99%. Despite the tool’s accuracy, we know false positives occur and in testing, it is approximately 2% of the time.

• See the Complete AI Detection Accuracy Study

Quick overview showing Originality.AI’s accuracy on GPT-4 and comparing its detection accuracy to other AI detection tools.

Originality.AI Detection Accuracy on GPT-4

Below is a confusion matrix used to test the accuracy of an AI detector against a set of AI generated and human generated text…

• True Positive – AI detector correctly identified content as AI.
• False Negative – AI detector incorrectly identified AI content as Human.
• False Positive – AI detector incorrectly identified human content as AI.
• True Negative – AI detector correctly identified human content as AI.

Below are the results of a study comparing the Accuracy and False Positives of Originaltiy.AI vs Other AI Detectors

Are AI Detectors Easy to Trick?

A lot of the existing AI detectors have a very easy way to trick them. Simply swapping out some words using a Paraphrasing tool like Quillbot results in the ability to bypass detection at all detectors except Originality.AI. Originality.AI can detect if content whether it is AI or Original was paraphrased.

• AI Detection Accuracy
• Detecting when Paraphrasing Tools, like Quillbot, Have been Used

What is and is not a False Positive?

A false positive is when an AI detector incorrectly identified human-created content as being likely generated by an AI.

There is often some misunderstanding when it comes to false positives. For clarity here is how Originality.AI aims to identify AI vs Original content under different content creation scenarios…

• AI-Generated and Not Edited = AI-Generated Text.
• AI-Generated and Human Edited = AI-Generated Text.
• AI Outline, Human Written and heavily AI Edited = AI-Generated Text.
• AI Research and Human Written = Original Human-Generated.
• Human Written and Edited with Grammarly = Original Human-Generated.
• Human Written and Edited = Original Human-Generated.

So if the content was outlined by an AI, then some of the content was written by a human and finally edited/expanded on by an AI… Originality.AI aims to identify this as AI-generated. This is not a false positive.

Similarly and more obviously… if ChatGPT creates a piece of content and then someone painstakingly edits it but it still get identified as AI-generated – this is not a false positive.

Should AI Editing be Treated as AI or Original?

This is a tricky question and one that not everyone agrees with. Still, our approach at Originality.AI is that if the use of AI would result in the modified piece of content passing plagiarism, when compared to the original unedited text, then it should be identified as AI-generated.

How Should AI Detectors Be Used?

So in a world where the rate of false positives is not (or will ever be) 0%, how should AI detectors be used?

Because AI detection does not provide a proveable output for each piece of text… meaning no AI detection can say with proveable certainty that this is how we KNOW that your content was created with AI.

Therefore it is best to…

1. Not have a HARD rule that every piece of content MUST meet a certain threshold.
2. Look at a series of articles from the same writer to identify writers that are likely using AI vs those that are not.

Many Originality.AI customers using this strategy have been able to successfully identify writers that were using AI even though they had been asked not to. Additionally, the Originality.AI customers using this approach were confident they could safely ignore a suspected false positive.

For academic disciplinary action, AI detection scores alone are simply not enough.

• How to Think About AI Detection Scores

If Detection Scores Aren’t Perfect – Why Bother With Detection Tools At All?

Some might question if it is responsible to have an AI detection tool if it is not perfect. At originality.AI, we are confident in the detection rates of our tests and the additional steps we have taken to try and reduce/manage“False positives, including the free tools we offer”.

In a world where AI content is allowed to run wild unchecked, the impact on many of us would be significant. For example…

• For Web Publishers – If you are concerned about staying on the right side of Google’s guidance with AI-generated content “Using automation—including AI—to generate content with the primary purpose of manipulating ranking in search results is a violation of our spam policies.” then the use of AI detectors is needed to help protect your site.
• If you are happy to be publishing AI-generated content then you don’t need AI detectors or expensive writers.
• For Content Marketing Agencies – If AI detectors are not used, it will be impossible to communicate to clients who are asking to prove that the content you are receiving is not AI generated.
• For Writers – If AI detectors are not used then you would be competing against an army of ChatGPT users and struggle to prove your contents worth.
• For Academia (Students/Professors/Institutions) – If AI is allowed to be used unchecked, it will negatively impact the value of the degrees issued.

ChatGPT and other AI tools like HuggingChat are here, there is no going back and everyone involved with writing words will need to adapt to this new world. This doesn’t touch on the societal impact on the inability to distinguish between human-created and mass AI created propaganda content.

Multiple Examples of Students Being Falsely Accused of Using AI

The rise of ChatGPT and AI detectors has led to some unfortunate situations in academia.

Here are several examples:

1. https://www.reddit.com/r/GPT3/comments/10qfyly/my_professor_falsely_accused_me_of_using_chatgpt/
2. https://www.reddit.com/r/ChatGPT/comments/12ppt5w/my_teacher_has_falsely_accused_me_of_using/
3. https://www.reddit.com/r/ChatGPT/comments/11fj3tx/accused_of_using_ai_on_my_high_school_social_paper/

To be clear… at Originality.AI we don’t believe that an AI detection score alone is enough for disciplinary action and Originality.AI is built to focus on supporting writers, marketing agencies, and website publishers.

What do You Do If Falsely Accused – As a Student

If you truly created (without AI involvement) a piece of text and you are being accused of having created it with AI, here are steps you can take…

1. Take your time – A knee-jerk reaction can often make the situation worse. Take the time you need to respond in an intelligent way.
2. Collect Relevant Information
3. Review your school’s academic integrity policy as it relates to the use of AI.
4. Collect any working notes, revision history, etc that demonstrate you put a lot of effort into the work.
5. Ideally, you created the text in a Google Document and you can use the Originality.AI free Chrome extension to show the documents entire creation process – https://originality.ai/free-ai-content-detector-chrome-extension/
6. Find relevant content showing the detection accuracy and false positive occurrences of tools such as – https://originality.ai/ai-content-detection-accuracy/
7. Check your previously submitted work to show your current work is similar to previously submitted work.
8. Have Empathy for Your Professor – They are needing to maintain the academic integrity of their institution in the face of a wave of students using AI to cheat and the tools they have to fight it are imperfect.
9. Send a Polite Response to Your Teacher – Communicate the information you have compiled in a polite/respectful way that you are confident this is a case of a false positive.
10. Escalate the issue to your institution’s administration if not resolved with your teacher.

What do You Do If Falsely Accused – As a Writer

If you are a writer and you have been falsely accused of having AI create you

1. Collect Relevant Information
2. Did your writing agreement state that AI was not to be used?
3. Was a certain AI detection score and tool agreed to (not a recommended way to manage the misuse of AI).
4. Ideally, you wrote the article inside of a Google Document and you can easily share the visualization of the creation process – https://originality.ai/free-ai-content-detector-chrome-extension/.
5. Find previously submitted and similar work running it through an AI detector.
6. Provide research and guidance on how to handle AI detection scores.
7. AI Detection Accuracy
8. How to Use AI Detection Scores
9. Share the relevant information with whoever is accusing you incorrectly of using AI. If you can share the information below it should be able to resolve any issue of a false positive…
10. Demonstrate your other similar work was Original.
11. Show the creation process (using our free ChatGPT detector Chrome extension) of both your previous content and the content in question.
12. Share research (AI detection accuracy & recommendations on how to use AI detection scores).

How to Avoid False Positives

Here are 8 tips that can help resolve false

1. A detection score of 60% Original and 40% AI is not a false positive. It correctly predicted with 60% confidence that your content was Original. The meaning of the score is not that 60% was Original and 40% was AI; it means it is 60% confident that the content is original.
2. Create all articles in Google Docs (whenever possible) so that you can use our free Chrome Extension to help you PROVE your content is Original.
3. Editing AI-written content is not a false positive; it is a true positive.
4. Having AI edit your work is not a false positive; it is a true positive.
5. When any amount of AI touches the content, it can cause the entire article to be flagged as AI.
6. "Cyborg" writing, where a lot of AI tools are used to create an outline, suggest edits, and optimize the content, can increase the chance of a higher AI score.
7. For this reason, we created a free content optimizer tool similar to SurferSEO or MarketMuse but 100% free that does not use AI to reduce the chance of a false positive occurring.
8. Strange formatting can reduce the accuracy of the detector tools, causing an increase in false positives or false negatives.
9. The shorter the text, the less accurate the detection score. We recommend at least 100 words be checked.

"False positives in AI detection are a BIG deal and will not go away with the use of generative." AI continues to climb. Our hope is that this article will help people understand the limitations of detection tools and share strategies on their appropriate use and how to prove your work's Originality.