We analyzed 29,000 reviews about 784 companies at Capterra, a site for reviews about B2B software. Our AI detector examined the text of reviews to find how much content was likely generated by humans or AI.
Summary of Key Findings:
AI in Capterra Reviews Study Findings
More than 30% of Reviews on Capterra, since the launch of ChatGPT, are suspected of being AI generated
Our AI detector examined the text of reviews to find how many of them were likely generated by humans or AI. We calculated what percent of reviews posted each month were AI-generated and tracked how that percentage changed over time. The amount of AI-generated reviews grew year after year as GPT-2 and GPT-3 were introduced. Then there was an immediate spike when ChatGPT became available on November 30, 2022. The 14.8% of reviews were AI-generated in November 2022 and then shot up to 29.7% in December 2022. The first 11 months of 2002 averaged 15% of reviews being from AI, while December 2022 through September 2023 averaged 33.6%, an increase of 224%.
Note that the ~5% of reviews that were identified as AI generated before 2020 represent false positives that occur with AI detectors. Reference our accuracy study for more details - https://originality.ai/ai-content-detection-accuracy/
High Ratings of 4 or 5 Were 2.1 Times More Likely to Be AI-Generated
Capterra ratings are on a scale of 1 to 5. We found that 17.9% of reviews with a rating of 4 or 5 were written by AI, and that 8.4% of reviews with ratings of 1 to 3 were after the launch of ChatGPT. This difference made higher ratings 2.1 times more likely to have AI-generated reviews than lower ratings.
Using AI to generate text for reviews presents challenges for everyone trying to use those reviews to make decisions, whether for personal or business uses. How do you feel about a review that was AI-generated? There could be times when it could be appropriate to use ChatGPT to improve readability, but is that what is happening in more than 30% of Capterra reviews? Do you want to feel like you’re in a Turing test every time you read a review? The findings in this study show how detecting AI can change how we use sites like Capterra.
We believe that it is crucial for AI content detectors reported accuracy to be open, transparent, and accountable. The reality is, each person seeking AI-detection services deserves to know which detector is the most accurate for their specific use case.