Do AI Detectors Work? OpenAI says “in short no” but the reality is that, although they are not perfect, they do “work” and if you think they don’t we have a challenge for you that will benefit charity!
The usefulness of AI detectors depends on your use case, if 95%+ AI detection accuracy is enough then they definitely “work”. However, if perfection is required then they are not the right tool for you. They will never be “perfect” and for this reason, we have always recommended against them being used for academic discipline.
But we believe the societal costs that would result from undetectable AI-generated text are too significant for over-simplified comments stating AI detectors don’t work, such as OpenAI’s above, backed up with no data.
So…
We are issuing a “Do AI Detectors Work” Challenge for Charity and specifically asking OpenAI to participate and stand behind their statement “Do AI Detectors work? In short, no”!
If you also think AI detection tools do not work then here is your chance to expose them and benefit a good cause!
Announcing The Do AI Detectors Work Challenge for Charity
The idea is that we will run a dataset, newly created for this test, of AI-generated text and human-written text through Originality.ai.
For each piece of writing Originality incorrectly identifies we will donate to charity, for each one Originality.ai predicts correctly the challenger will donate to charity.
Interested? Contact Us and let’s set it up!
When text is entered into Originality you receive a probability that the text is Original (human-created) and the probability that the text is AI (AI-written).
If the text was human-written and Originality.ai identifies a greater than 50% chance that it is Original that is a correct prediction.
So either human-created or AI-created text can be entered and Originality can predict if the content was Original or AI-generated so it produces 4 possible outcomes.
If you want to run a modified version (smaller data set etc.) we are open to that, contact us and we would be happy to work with anyone to set up a fair test!
Here is our idea…
10,000 Text Record Dataset Tested
If you think AI detectors do not work and want to take us up on this challenge then please contact us.
Plus it is for a good cause!
This is an invitation for OpenAI and we would be happy to have others join in on the fun and benefit charity.
Our hope is to also provide some understanding of the effectiveness of AI detectors and their limitations!
Below we will provide some additional context on how AI content checkers work and their limitations based on our testing.
AI detectors may each use some combination of these detection models…
Further Reading:
We have an incredibly in-depth study that looks at our and other companies' AI detector accuracy. But in short, not all detectors are the same and their usefulness depends on your use-case/dataset.
The accuracy claims made by any detector that doesn’t provide data to support that claim should be taken with a grain of salt.
No detector is perfect and all will have some % of false positives (a false positive is when the detector incorrectly thinks a human-written article was AI-generated) and false negatives (the detector incorrectly thinks an AI-written article was human).
Here is the accuracy of Originality.ai on GPT-4 generated content (99%+ accurate) with a 1.5% false positive.
Here are the results of testing several AI detectors on an adversarial dataset (dataset of AI-generated content meant to be undetectable) that we have open-sourced.
Full results including the dataset used can be checked here:
https://originality.ai/blog/ai-content-detection-accuracy
Additionally at the link above you will find an AI detector efficacy analysis tool we open-sourced that will allow you to run your own dataset across multiple detectors to determine their effectiveness for your use case. This tool includes a full statistical analysis including a confusion matrix and reporting the F1, Precision, Recall (True Positive Rate), Specificity (True Negative Rate), False Positive Rate, and Accuracy.
We do not like and have always recommended against the use of Originality.ai, or any AI detector, being used for academic dishonesty and disciplinary actions.
A lot of the confusion related to AI detectors stems from people looking to use them the same way that plagiarism checkers are used.
Plagiarism checkers are able to provide an enforceable level of proof that text was copied. AI detectors can not do that.
For this reason, we are not supporters of academic discipline being issued based solely on the results of an AI detector.
If it were possible to bypass a detector easily then would they still serve a purpose? I think we would agree that they would be less useful. But the reality is that many of the bypassing methods that previously produced useful content are no longer effective. Now the reality is that writing tools that make AI-generated content undetectable produce output quality that is simply not useful for most people. If you then correct the grammar and odd word choices it typically becomes detectable again.
Here is a guide that looks at the different strategies people use to make undetectable AI content.
Quillbot or other paraphrasing tools used to be effective at bypassing Originality and other AI content detection tools but that is no longer an effective strategy at Originality.ai. See results for Quillbot and Paraphrasing Detection.
A score of 40% AI and 60% Original means the detector thinks there is a 60% chance the piece of content was written by a human and a 40% chance it was written by a machine.
It does not mean that 40% of the article is AI-generated and 60% is human-generated text. Although it can be very frustrating when you see a score of 40% AI on a piece of content that you know you wrote 100% yourself, this is not a false positive.
To understand if an AI content detector will “work” for your use case you first need to understand what level of accuracy you require.
If you require perfection then AI detectors will not work.
If over 95% accuracy and under 5% false positives is acceptable for your use case and the alternative of having no idea if content was AI or human generated then they will work for you.
To understand the exact efficacy of detectors for your use case you can use the open-sourced tool we provide and test your own dataset - https://originality.ai/blog/ai-content-detection-accuracy
If you think AI-generated text detectors do not work we offer you the chance to prove it and benefit charity! Please Contact Us!