Is GPT-4o-mini Content Detectable?

In July 2024, OpenAI released their latest AI model — GPT-4o-mini. In their release, OpenAI describes GPT4o-mini as the “most cost-efficient small model.”

As the costs of AI content generators and AI writing tools continue to drop and as these tools become increasingly accessible, it’s more necessary than ever to understand the accuracy of AI detectors.  

This brief study looks at 1000 GPT-4o-mini generated text results to find out whether the Originality.ai AI Detector can detect GPT-4o-mini.

TL;DR — Is GPT-4o-mini AI-Generated Content Detectable?

• Yes — GPT-4o-mini generated-text is detectable with exceptional accuracy across the Originality.ai AI detection models.
• Results Overview — 3.0.0 Turbo demonstrated 99.9% accuracy, 1.0.0 Lite had 99.6% accuracy, and 2.0.1 Standard had 99.4% accuracy.

Try the Originality.ai AI Detector. Then, learn about AI content detection accuracy and Originality’s exceptional performance in a meta-analysis of six third-party studies. 

Note: Standard is now retired. For more information on our AI detection models, check out our guide on which model is best for you.

Dataset

To evaluate the detectability of GPT-4o-mini, we prepared a dataset of 1000 GPT-4o-mini-generated text samples.

The Method: gathering AI-generated text data

For AI-text generation, we used GPT-4o-mini based on three approaches:

1. Rewrite prompts: The first approach generated content by providing the model with a custom prompt and articles (probably generated by LLMs) as a reference to rewrite from (450 Samples).
2. Rewrite human-written text: The second approach involved generating content with the aim of finding out whether AI rewrites of human-written text could bypass AI detection. The samples for this approach were fetched from an open-source dataset (325 Samples).
   1. One-Class Learning for AI-Generated Essay Detection
      1. Paper: https://www.mdpi.com/2076-3417/13/13/7901
      2. Dataset: https://github.com/rcorizzo/one-class-essay-detection
3. Write articles from scratch: The third approach, generated articles from scratch based on topics ranging from fiction to non-fiction, such as history, medicine, mental health, content marketing, social media, literature, robots, the future, etc. (225 Samples).

Evaluation

To evaluate the efficacy, we used the Open Source AI Detection Efficacy tool that we released:

• AI Detector Used in this Study
• AI Detector Efficacy Testing Article
• Open Source AI Detection Efficacy Testing Tool

A brief overview of the Originality.ai AI detection models

Originality.ai has three models, 3.0.0 Turbo,  2.0.1 Standard, and 1.0.0 Lite, for AI text detection.

• 3.0.0 Turbo — If your risk tolerance for AI is ZERO! It is designed to identify any use of AI, even light AI.
• 2.0.1 Standard — A balanced model that is a great option if you are okay with slight use of AI (i.e., AI editing).
• 1.0.0 Lite — If you permit light AI editing (such as Grammarly’s spelling or grammar suggestions).

For additional information on each of these models, check out our AI detector and read our AI detection accuracy guide.

Evaluating the AI detection models

The open-source testing tool returns a variety of metrics for each detector you test, each of which reports on a different aspect of that detector’s performance, including:

• Sensitivity (True Positive Rate): The percentage of time the detector identifies AI correctly.
• Specificity (True Negative Rate): The percentage of time the detector identifies human-written content correctly.
• Accuracy: The percentage of the detector’s predictions that were correct.
• F1: The harmonic mean of Specificity and Precision, often used as an agglomerating metric when ranking the performance of multiple detectors.

For a detailed discussion of these metrics, what they mean, how they're calculated, and why we chose them, check out our blog post on AI detector evaluation. For a succinct snapshot, the confusion matrix is an excellent representation of a model's performance.

Below is an evaluation of all these models on the above dataset. 

Confusion Matrix

Results of the Evaluation :

For this smaller test to identify the ability of Originality.ai’s AI detector to detect GPT-4o-mini content, we reviewed the True Positive Rate or the percentage (%) of time that the model correctly identified AI text as AI out of 1000 samples of GPT-4o-mini content. 

1.0.0 Lite:

• Recall (True Positive Rate) = 99.6%

2.0.1 Standard:

• Recall (True Positive Rate) = 99.4%

3.0.0 Turbo:

• Recall (True Positive Rate) = 99.9%

Final Thoughts

Overall, Originality.ai continues to demonstrate an outstanding capability to identify AI-generated content, including the latest releases of AI models such as OpenAI’s GPT-4o and GPT-4o-mini. 

Each of Originality.ai’s AI detection models detected GPT-4o-mini with exceptional accuracy from 3.0.0 Turbo with 99.9% accuracy to 2.0.1 Standard with 99.4% accuracy, and 1.0.0 Lite with 99.6% accuracy.