AI Studies

AI-Generated Capterra Reviews Spike After ChatGPT Launch

We analyzed 29,000 reviews about 784 companies at Capterra, a site for reviews about B2B software. Our AI detector examined the text of reviews to find how much content was likely generated by humans or AI.

We analyzed 29,000 reviews about 784 companies at Capterra, a site for reviews about B2B software. Our AI detector examined the text of reviews to find how much content was likely generated by humans or AI.

Summary of Key Findings:

  1. More than 30% of reviews on Capterra since the launch of ChatGPT are likely to be AI-generated. This represents an increase of 224%.
  2. High ratings of 4 and 5 were 2.1 times more likely to have AI-generated reviews.

Study Method and Data:

  1. The text from 29,000 Capterra reviews that were at least 100 words long were rated by Originality.AI’s AI detector as human or AI generated.
  2. We tracked the percentage of reviews rated as AI-generated over time.
  3. The product rating given along with each review was analyzed along with whether the review was written by a human or AI. The relationship between ratings and AI-generated text was analyzed. 

AI in Capterra Reviews Study Findings

More than 30% of Reviews on Capterra, since the launch of ChatGPT, are suspected of being AI generated

AI in Capterra Reviews Study Findings

Our AI detector examined the text of reviews to find how many of them were likely generated by humans or AI. We calculated what percent of reviews posted each month were AI-generated and tracked how that percentage changed over time. The amount of AI-generated reviews grew year after year as GPT-2 and GPT-3 were introduced. Then there was an immediate spike when ChatGPT became available on November 30, 2022. The  14.8% of reviews were AI-generated in November 2022 and then shot up to 29.7% in December 2022. The first 11 months of 2002 averaged 15% of reviews being from AI, while December 2022 through September 2023 averaged 33.6%, an increase of 224%.

Note that the ~5% of reviews that were identified as AI generated before 2020 represent false positives that occur with AI detectors. Reference our accuracy study for more details - https://originality.ai/ai-content-detection-accuracy/

High Ratings of 4 or 5 Were 2.1 Times More Likely to Be AI-Generated

High Product Rating Reviews Twice as likely to be ai generated on Capterra

Capterra ratings are on a scale of 1 to 5. We found that 17.9% of reviews with a rating of 4 or 5 were written by AI, and that 8.4% of reviews with ratings of 1 to 3 were after the launch of ChatGPT. This difference made higher ratings 2.1 times more likely to have AI-generated reviews than lower ratings.

Summary

Using AI to generate text for reviews presents challenges for everyone trying to use those reviews to make decisions, whether for personal or business uses. How do you feel about a review that was AI-generated? There could be times when it could be appropriate to use ChatGPT to improve readability, but is that what is happening in more than 30% of Capterra reviews? Do you want to feel like you’re in a Turing test every time you read a review? The findings in this study show how detecting AI can change how we use sites like Capterra.

Turing Test

Jonathan Gillham

Founder / CEO of Originality.ai I have been involved in the SEO and Content Marketing world for over a decade. My career started with a portfolio of content sites, recently I sold 2 content marketing agencies and I am the Co-Founder of MotionInvest.com, the leading place to buy and sell content websites. Through these experiences I understand what web publishers need when it comes to verifying content is original. I am not For or Against AI content, I think it has a place in everyones content strategy. However, I believe you as the publisher should be the one making the decision on when to use AI content. Our Originality checking tool has been built with serious web publishers in mind!

More From The Blog

AI Content Detector & Plagiarism Checker for Marketers and Writers

Use our leading tools to ensure you can hit publish with integrity!