AI Studies

AI Content in Amazon Reviews

We analyzed almost 2,000 Amazon product reviews to answer that age-old question: ‘Did an AI write that 5-star review? ’Specifically, we investigated the growing trend of AI-generated content in these reviews, its relationship with known quantities, and its ultimate influence on shopping decisions.

Jonathan Gillham

We looked at 26,000 Amazon product reviews to answer the question: ‘Did an AI write that review?’

Summary of Key Findings

  • 400% increase in AI generated review since the launch of Chat GPT.
  • With the advent of Chat GPT in November 2022, there was an explosion of the use of AI-generated content in product reviews by approximately 400%
  • Extreme reviews are 1.3 times more likely to be AI generated.
  • Reviews with strong bias are more likely to be written with AI assistance
  • Verified reviews are 1.4 times less likely to have AI Content
  • The good news is that AI generated reviews are more likely to be from unverified sources and regarded as unhelpful. The bad news is that might not be the case for much longer.
  • Users Intuitively Find AI Generated Reviews Less Helpful
  • There is a small but significant negative correlation between the helpful scores of a review and its probably AI Content.

Methodology

Over 26,000 Amazon product reviews were randomly selected to generate the source dataset. These records were cleaned using standard processes, which included removing all customer-sensitive and identifying information. A statistically representative subset of 2,000 records was processed with Originality.AI’s AI Detection API. The most accurate AI detector to analyze text data for the likelihood of AI Content and returns a probability score of 0 to 1.

Findings

Since Chat GPT launched AI generated reviews increased by 400%

AI-enhanced text generators pre-existed Chat GPT. GPT-3 was released in June 2020, quickly followed by the writing aid tools that leveraged it. However, since the launch of Chat GPT, text generators progressed from limited, niche applications to more widespread usage. Trend analysis of the dataset indicates that on average, the number of product reviews with 50% or more AI Content has grown since the average annual value of 0.02 in 2022. As at publishing date, this value has increased by approximately 400%. There are no signs of slowing down.

Ai Usage in Amazon Reviews

Extreme Reviews are 1.3x More Likely to be AI Generated

There are two kinds of reviews: The extreme 1-ers and 5-ers who either love or hate a product passionately and want everyone to know this. Or the Chill Trio – 2s, 3s, and 4s. As it turns out, they’re Chill in more ways than one as these reviews appear less likely to rely on AI text generators. According to our findings, AI content is approximately 1.3 times more likely to be detected in Extreme reviews than Moderate reviews.

Visual Analysis of Rating Severity vs ai Content

Verified Reviewers are 1.4x Less Likely to be AI Generated

Visual Analysis of is Verified vs ai Content

There are some bright spots. Analysis shows that Verified reviewers are roughly 1.4 times less likely to be AI generated than non-verified reviewers. Amazon.com values the integrity of the review process and goes to great lengths to safeguard it. However, not all ecommerce websites have Amazon’s resources.

Users Intuitively Find AI Generated Reviews Less Helpful

Standardized Helpfulness Count Vs Ai Content Probability

Another optimistic discovery came from analyzing the helpfulness score of a review (eg. ‘20 people found this review helpful’) that is used to rank reviews. It appears that people are more likely to be instinctively biased against AI content. Or to be precise, there is -0.06 correlation on average between standardized helpfulness and likelihood of AI Content. And yes, that’s supposed to be the good news. The bad news is that this figure doesn’t factor how potential bot-voting skews that statistic or if reviews that have been around longer get more votes.

Summary: Amazon Ai Generated Reviews

Further analysis is still being done. It will take some time to build a large enough dataset to answer all the burning questions with statistical confidence. But this is enough to get us thinking. There’s nothing wrong with the use of AI Content, especially in a free public service like sharing your genuine opinion on a product. But that’s assuming that all those reviews are free and genuine, and not… 😷motivated by vested interests, competition, and paid reviews 😷. 

Finally, ask yourself: would it make a difference to your purchasing decisions if you knew that the review that persuaded you to click on ‘Add to Cart’ came from a heartfelt human anecdote… or from the articulate logic of a well-trained machine? 

The Github repository of the notebooks, source codes and the cleaned datasets for these experiments are available here

If you have any questions regarding this study please contact us.

Jonathan Gillham

Founder / CEO of Originality.AI I have been involved in the SEO and Content Marketing world for over a decade. My career started with a portfolio of content sites, recently I sold 2 content marketing agencies and I am the Co-Founder of MotionInvest.com, the leading place to buy and sell content websites. Through these experiences I understand what web publishers need when it comes to verifying content is original. I am not For or Against AI content, I think it has a place in everyones content strategy. However, I believe you as the publisher should be the one making the decision on when to use AI content. Our Originality checking tool has been built with serious web publishers in mind!

More From The Blog

AI Content Detector & Plagiarism Checker for Serious Content Publishers

Improve your content quality by accurately detecting duplicate content and artificially generated text.