AI Vs Human Content Detection Score Meaning

Unlock the full potential of your content creation process with our guide on interpreting Originality.AI’s AI vs Human score. Whether you’re a writer, editor, marketing agency, or website publisher, this article will help you understand and align on what fits for your content marketing strategy.

I will try and answer all of these questions the best I can below…

• What does the AI score mean?
• How much “AI” is acceptable for Google?
• Just tell me… did this piece of content get created by AI? Yes or No!
• How sure is the Originality of its score?
• I created this content myself but it still scored as some % AI… why?

How is Originality.AI Trying to Help?

We are working to set the modern, trusted standard for determining the originality of a piece of content. We want to be your go-to 3rd party helping ensure writers, editors, marketing agencies, and web publishers all can trust in how content has been created. Was it copied, was it spun by a paraphrasing tool, did it get created by an AI tool like ChatGPT, or did a human write it? We are here to help!

Content created by AI has a place both now and into the future but I believe most of us do not want to pay the same rate for content that was generated in 5 minutes by AI compared to a thought leadership piece created by a true expert.

We want to ensure the content creation process is transparent and fair for everyone involved in the process.

Plagiarism Checking and AI Detection Should Not Be Treated the Same!

Checking for plagiarism is widely understood, it has been done online for decades. It is a very clear process where if a section of text is copied from somewhere else then it is plagiarized. However, checking and interpreting the results of AI detection is a much more nuanced activity. The results are not as “binary” as plagiarism detection meaning there is more room for interpretation.

• If an article is 5% Plagiarized that means 5% of the article text is likely copied from another source, however.
• If an article has an AI score of 5% and a Human score of 95% there is a 95% chance that the article was human-generated (NOT that 5% of the article is AI generated)

This document is meant to help get everyone to use and understand the AI detection scores correctly.

AI Content Detection Accuracy:

Across an exhaustive test of 10,000 pieces of GPT-3 generated content along with a human-generated control group Originality.AI was over 94% accurate.  For ChatGPT and GPT-3.5 you can see the results across 20 generated articles:

https://originality.ai/can-gpt-3-5-chatgpt-be-detected/

The tool is 94%+ accurate with a few false positives and negatives which is excellent… however it is not at an “enforceable” level since there will be incorrectly identified articles.

So… given that it is very accurate but not perfect how should we interpret the results?

How to Interpret Results:

Let’s look at how to interpret results.

It is important to remember 2 things at this stage…

1. % AI and % Human represents the probability that the article was created by AI or Human NOT how much of the article was AI and how much was Human.
2. The accuracy is great but not perfect (94%+)

So given these 2 points the best way to review the results is to look at a series of results from a single source (writer or agency).

Here is how to think about the scores from a group of writers.

Recommended AI vs Human Score Thresholds:

It is important to remember, as smart as the development team that built Originality.AI’s AI detection tool, we are not Google.

I built Originality because I wanted publishers to be able to make intelligent risk-based decisions when it comes to publishing potentially AI-generated content.

So based on your risk tolerance, the writing team and the website it is being published on, your decision on the allowable threshold of AI will be different.

Here are some recommended thresholds to try and help get Writers, Marketing Agencies, and Publishers all on the same page.

We are seeing companies choose between 4 AI strategies and here would be the recommended Human vs AI scores to aim for across a sampling of content…

Strategies & Suggested Targets:

1. No AI Ever!: For people looking to ensure all the content on their site is Human Generated…
2. Human Average: 90%+
3. Human Minimum: 65%
4. AI Used for Research: For websites who do not “ban” AI tools and allow their writers to use them for research, brief creation, etc.
5. Human Average: 75%
6. Human Minimum: 50%
7. Heavily Edited AI: For websites looking to leverage AI for efficiency but are scared of Google detecting their content as AI-generated.
8. Human Average = 60%
9. Human Minimum = 50%
10. Ethical AI Content: For websites looking to scale content with AI but have an editor for fact-checking.
11. No Target…Human Minimum of 0% is acceptable

This is a good point to put a reminder that we are not Google. Our AI detection likely works in a more robust but potentially less efficient way than anything Google could roll out at scale. Our teams’ belief is that if our tool is unable to tell if the content is AI then likely Google and their potentially smarter (but by necessity more efficient) approach would get a similar result.

How is Our AI Tool Different:

Our AI takes a more holistic and far more intensive approach to analyze an article to determine if it was AI generated than anything else we have seen.

Here is an in-depth article looking at how our tool works. It is our own AI that was trained on an incredible amount of GPT-3, and GPT-3.5 content and is able to accurately identify patterns in AI content across an entire article – https://originality.ai/how-does-ai-content-detection-work/

Our approach is a far more intensive/accurate way to look at an article than the free AI detection tools that rely on either…

1. A linear probabilistic model – These tools use OpenAI’s model to predict what the next word would be based on the NLP model. If enough of the next words in a piece of content fit within what the model would predict then it is likely AI. This is easy to fool with the changing of a few words.
2. “Burstiness” and “Perplexity” – These tools rely on the fact that AI content is typically consistently created without much variation. They are susceptible to being tricked and resulting in many false positives.

What Part is AI?

We appreciate the valid question of “what part is AI” this is going to be answered with a highlighting solution that identifies blocks of text based on how likely that block of text was generated by AI.

This will be live shortly.

Alternative to Using Originality.AI or AI Detection

I understand the desire to get a yes or no answer to the question of whether or not a piece of content was created by AI but like all things, in SEO the answer is unfortunately nuanced.

The alternative to ignoring the presence of AI is inviting its use on your site. Depending on your editing process that is not necessarily a bad decision.

Summary and Key Takeaways:

• The AI vs Human score says the probability that our AI thinks your content was generated by AI or generated by Humans.
• A score of 10% AI 90% Human
• Means – Originality.AI gives the content a 90% probability of being Human and a 10% probability of being AI generated
• Does Not Mean – 90% of the article is human and 10% of the article is AI
• Publishers are using a high human score as a good measure of high Originality even when they are confident a human created it.
• The tool is 94%+ accurate on GPT-3, GPT-3.5, and ChatGPT. However, it is not perfect and there are some false positives and false negatives
• It is far better to review a series of articles and make a call on a writer/service compared to passing judgment on a single article.
• Length Matters – The longer the article the more confident the scoring becomes.

‍