Grammarly AI Content Detector Review: Easy to Bypass

As AI content generation continues to advance, more brands are adding AI-based features and AI tools to their offerings.

One brand that’s recently gone down this avenue is Grammarly. In August 2024, Grammarly released an announcement that it’s adding an AI content detector to its suite of tools. This news comes despite Grammarly’s previous stance that AI detection tools are ‘unreliable.’ 

In this review, we’ll test the Grammarly AI Content Detector to see if it’s reliable.

In This Article

• Find out what Grammarly is
• Discover why there’s interest in tools like Grammarly's AI detector
• Review our AI content detection tests with Grammarly and the Originality AI Detector
• Get a final verdict on whether Grammarly is a reliable tool

What Is Grammarly?

Grammarly initially launched into the mainstream as a grammar-checking tool, offering users a free plugin that helped them ensure their grammar was correct, their word usage was as effective as possible, and the structure of their content matched their intent.

Since then, the tool has become a go-to option for marketers, small business owners, and writers.

Despite their apparent distrust of AI content detectors as ‘unreliable’ (see the above screenshot), Grammarly has decided to create their own tool. Hopefully, this is a sign that the company is coming to an understanding of the importance of AI detection.

Why are people interested in Grammarly's AI Detector?

There is no doubt that AI content generation has its benefits. It can be an incredible tool for creating content briefs or helping with content ideation.

However, AI text also creates problems, including plagiarism, a lack of transparency, and problems with potentially getting marked as spam by Google.

With that in mind, industry leaders, business owners, and marketing experts are looking for a reliable AI detector to gain clarity for content production and content consumption.

Grammarly AI Detector Review With RAID Dataset

In our evaluation of the Grammarly AI Detector using the RAID AI detection accuracy study dataset, it is clear that the detector is highly susceptible to bypassing techniques, especially when tested with adversarial attacks.

The RAID dataset, designed to test the robustness of AI content detectors, showed critical weakness in Grammarly’s AI detection capabilities.

Grammarly’s Performance

• Poor Performance: Grammarly's AI detector scored an F1 of 0.364 and a low recall of 0.222, missing most AI-generated content.
• Easily Bypassed: Simple text alterations, such as paraphrasing, alternative spellings, and formatting changes, can evade Grammarly's detection.
• Limited Scope: Struggles with AI-generated content from models beyond GPTs or Llama, showing a narrow detection capability.
• Accuracy: 22.22%

Originality.ai’s Performance

• Strong Performance: Originality.ai scored an F1 of 0.918 and a high recall of 0.848, effectively detecting most AI-generated content.
• Highly Accurate: With a perfect precision of 1.0, Originality.ai’s AI checker accurately detected AI-generated content with an Accuracy of 84.84%
• Wide Detection Capability: Performed consistently across different adversarial attacks and AI models, showing strong robustness against a wide range of AI-generated text.

Dataset used for evaluation

The RAID dataset (Robust AI Detection) is a benchmark specifically designed to evaluate the robustness of AI-generated text detectors.

It includes a wide variety of adversarial attacks and multiple AI models making it an effective tool to test a detector's ability to identify manipulated AI-generated content. The dataset's diversity in text manipulations and generative models helps expose weaknesses in AI detection tools, highlighting their susceptibility to adversarial techniques.

• Adversarial Attacks: alternative_spelling, article_deletion, insert_paragraphs, number, paraphrase, perplexity_misspelling, synonym, upper_lower, whitespace
• Models: chatgpt, cohere, cohere-chat, gpt2, gpt3, gpt4, llama-chat, mistral, mistral-chat, mpt, mpt-chat
• Total number of samples selected: 99 (11 from each kind of adversarial attack x model)

The verdict: Grammarly is easy to bypass with the RAID dataset

In conclusion, Grammarly’s AI detection might be able to catch obvious instances of AI-generated text. However, it’s easy to bypass with minimal effort. On the other hand, Originality.ai’s AI detector demonstrated a strong, robust performance as a more reliable AI content detection tool.

How We Tested Grammarly’s AI Detector With 3 ChatGPT Content Samples

As part of this review, we tested Grammarly’s AI detector with three different forms of AI content.

Step 1: We started by building out detailed prompts for each test before posting them into ChatGPT, one of the most frequently used AI content generators.

Step 2: From there, we took the exact copy output from ChatGPT for each test and pasted the results into Grammarly’s AI content detector (without making any changes) to help show whether or not the tool can detect AI-generated content.

Step 3: For comparison, we also took that exact same copy and pasted it into the Originality.ai AI content detector, allowing us to see how each tool compares with the same content input.

The three types of content used in this test include:

• Blog posts
• Research papers
• Promotional emails

AI Content Sample 1: Blog Post

Prompt: "Write a concise blog post on the benefits of investing in a summerhouse as a garden office, targeting professionals and business owners.

Key points:

1. Introduction: Introduce the idea of a summerhouse garden office and its rising popularity.
2. Productivity and Work-Life Balance: Explain how a dedicated garden office improves focus and helps maintain work-life boundaries.
3. Customization and Comfort: Highlight the ability to personalize the space for comfort and productivity.
4. Property Value and Aesthetic Appeal: Discuss the potential to increase property value and enhance garden aesthetics.
5. Cost-Effectiveness: Compare setup costs to traditional office expenses, noting long-term savings.
6. Environmental and Health Benefits: Mention reduced commuting and health perks from being closer to nature.
7. Conclusion: Summarize the benefits and encourage consideration of a summerhouse office.

Keep the tone professional, clear, and persuasive."

Originality.ai

Originality.ai AI Detector — 2.0.1 Standard Score: 99% AI & 1% Human-written

Grammarly AI

Grammarly AI Score: 99% AI & 1% Human-written

AI Content Sample 2: Research Paper

Prompt: "Write a research paper exploring the benefits of using a summerhouse as a garden office, targeting a scholarly audience.

Focus Areas:

1. Introduction: Outline the concept and relevance of a garden office in the remote work era.
2. Productivity and Work-Life Balance: Analyze the impact on productivity and maintaining work-life boundaries.
3. Customization and Comfort: Investigate how summerhouses can be tailored for ergonomic and comfortable workspaces.
4. Property Value and Aesthetics: Assess the effect on property value and garden design.
5. Economic and Environmental Benefits: Compare costs with traditional offices and explore environmental impacts.
6. Legal Considerations: Briefly examine relevant zoning and building regulations.

Structure the paper to provide a clear, data-backed analysis of each point."

Originality.ai

Originality.ai AI Detector — 2.0.1 Standard Score: 100% AI & 0% Human-written

Grammarly AI

Grammarly AI Score: 93% AI & 7% Human-written

AI Content Sample 3: Promotional Emails

Prompt: "Craft a promotional email for a summerhouse company offering a 10% discount on first orders over $4,000. The email should emphasize the benefits of adding a summerhouse to a garden, such as creating a functional and stylish space for work or leisure. Highlight the limited-time nature of the offer and encourage immediate action. The tone should be warm, persuasive, and focused on the value and quality of the summerhouses, with a clear call to action to use the discount code at checkout."

Originality.ai

Originality.ai AI Detector — 2.0.1 Standard Score: 100% AI & 0% Human-written

Grammarly AI

Grammarly AI Score: 71%% AI & 29% Human-written

Findings from our Grammarly AI detector analysis with ChatGPT content samples

From the results of this test, the Grammarly AI content detector demonstrated strong capabilities when it was faced with some of the most basic AI content detection, such as marketing content for blog posts.

However, when it came to slightly more complex content, such as promotional emails, where language patterns differ, and the text is shorter, Grammarly’s AI detector struggled compared to Originality.ai.

Searching for more AI detector reviews? Check out our guide on the Best AI Content Detection Tools Reviewed.

Want some additional insight into the efficacy of Originaltiy.ai’s AI detector, check out our meta-analysis of third-party studies.

Further Reading on Grammarly

Looking for further insight into Grammarly? Check out our collection of Grammarly resources:

• Grammarly AI Review
• Grammarly Plagiarism Checker Review
• Grammarly Readability Score Review
• Does Using Grammarly Make My Content Get Detected as AI-Generated?
• Does Grammarly Detect Generative AI and Plagiarism?

FAQs About AI Detection Tools and AI-Generated Content

Is Grammarly’s AI Content Detector free?

Grammarly’s AI content detector is free, and you don’t need a Grammarly account to use it. However, there is a word count limit for each submission.

What is Grammarly’s AI Detector?

Like other AI content detection tools, Grammarly’s AI detector is designed to identify AI-generated content and help users understand what content is human-written text and what isn’t.

Are AI-powered writing tools ethically okay to use?

When used sparingly, in the right context, and with transparency, AI writing tools have the power to be extremely beneficial, helping improve productivity and efficiency. However, when used to try and pass off AI work as your own, ethics comes into question.