GPTZero is an artificial intelligence (AI) content detection tool created by Princeton University student Edward Tian. He developed it for teachers after seeing the increased cases of “AI plagiarism.”

AI plagiarism refers to the use of an AI tool to create similar content based on existing information without proper attribution. Many educators and university lecturers have expressed concerns over AI content creation tools like ChatGPT and are worried students would use them on exams.

Tian positions GPTZero as the solution to this problem as it analyzes texts based on their perplexity and burstiness. Perplexity refers to the randomness of a text to a model or how well a language model likes a text. Meanwhile, burstiness refers to how uniform or constant the perplexity is over time compared to varied human-written text.

The app is currently in beta mode, with the creator planning to release a paper with accuracy stats based on data from student journalism pieces. 

Because AI content creation is becoming more common, it is up to apps like GPTZero to regulate AI use and protect the integrity of student work. Let’s learn more about this AI detection tool’s features and see if it is a viable solution to AI plagiarism.

Features

• Perplexity score: As mentioned earlier, perplexity is a randomness metric. It examines how good a language model (like ChatGPT) is at predicting the next words. Higher scores indicate unpredictability, implying that a text is human-made. Currently, GPTZero bases perplexity on the GPT-2 model. 

• Burstiness score: This metric looks at the distribution of sentences and determines whether there is uniformity or pattern. It’s based on the idea that humans tend to mix short and long sentences when writing. 

• • Highlighted AI-written portions: GPTZero’s latest update includes phrase detection. The app highlights which phrases are most likely written by an AI within a text. Additionally, GPTZero will show you which sentence has the highest perplexity score.

• API integration: The creator made GPTZero’s API readily available for anyone who wants to integrate it into their systems. You can also request customization support from the app’s engineers via email. 

Pros

• Free to use
• Can detect GPT-2 texts
• No sign-up needed

Cons

• Still in the beta stage
• Not entirely accurate
• Lacks features

Testing GPTZero’s Accuracy

To see if GPTZero’s accuracy holds up, we conducted a test using seven text samples generated by Jasper.ai, a popular AI writing tool. For comparison, we’ll show GPTZero’s results alongside Originality.ai’s. But before we show them, it’s important to explain how GPTZero presents results. 

GPTZero doesn’t use the probability format like many other AI content detection tools. It just provides a general statement of whether or not AI likely generates the text. Then, the app shows the perplexity score and the burstiness score. 

On the other hand, Originality.ai presents the results with percentages. It evaluates how much of a text is original and what percentage is most likely created by AI. Unlike GPTZero, Originality.ai is trained on several language models, namely GPT-3, GPT-3.5, and ChatGPT.

For comparison, we’ll show GPTZero’s perplexity score and statements. Here are the results.

Off the bat, you can tell that GPTZero’s results are quite confusing. The concept of perplexity, while defined clearly by the creator, is difficult to grasp in terms of the numbers shown. We can only go off from the statement that the higher the perplexity, the more likely the text is human-generated.

From our testing, we found that the perplexity score can go as high as 30,000. Aside from the general statements given by GPTZero, this is where we will base our analysis.

GPTZero could not catch 100% detection from any of the samples. The app would show this statement if it did: “Your text is likely to be written entirely by AI.” Meanwhile, Originality.ai showed 100% detection on samples #1 and #2. It also detects more than 90% on three more samples.

On the positive side, GPTZero does highlight phrases that are likely made by AI. Here’s an example:

Overall, GPTZero can detect instances of AI-generated content. However, the presentation of scores should be further improved for educators to truly grasp the gravity of a text’s AI plagiarism. In terms of accuracy and clarity, Originality.ai is the winner in this test.

GPTZero’s Potential

Since the tool is still in beta mode, it’s unfair to point out its drawbacks at this time. Instead, we can focus on where GPTZero is likely to go. For starters, the app is currently free, which makes it highly accessible to teachers.

The API is also available for those who request it. This means other developers can build on it and integrate it with their own systems. 

Edward Tian said from the beginning that the tool was built with academics in mind, unlike many AI content detection tools on the market that are designed for blog posts and marketing copy.

As the team tests GPTZero on more data from student work, the app’s accuracy is likely to increase. Combined with online plagiarism checkers, it can help improve the integrity of student essays and theses.

For now, educators should use the GPTZero sparingly while it is still being developed. As the app reminds users on its page, AI detection is just “one of the many pieces of a holistic assessment of student work.” You can also use plagiarism detection tools that scan documents against billions of public content online. 

Conclusion

GPTZero is a free and decent tool for catching AI plagiarism. However, it has a long way to go to catch up with competitors like Originality.ai, which detects not only AI content but also plagiarism.

However, it is important to note that GPTZero is still in beta mode. As the Princeton team continues to work on the app, we can expect its accuracy and features to improve. GPTZero’s metrics provide a decent starting point for teachers looking to detect AI plagiarism in the meantime.