75+ Claude AI Model Statistics in Q2 2024

In November 2023, the AI, and at a larger scale, the tech world was feverishly consumed by news of an incipient coup staged by the Open AI (the organization behind ChatGPT) board of directors. Ultimately, that coup failed to result in executive reform. Still, it did provide the perfect opportunity for one of its biggest competitors, Anthropic, to draw attention to its own LLM (large language model). As a result, on Tuesday, Nov 21st, 2023,  Anthropic announced grand updates to its AI model, Claude. Considering the enthusiastic market response, we will look at the many developing trends and statistics behind Claude AI. 

Claude, is an AI (artificial intelligence) based assistant developed by Anthropic using their research into HHH (Helpful, Honest, Harmless) AI applications. The assistant can be accessed through a chatbot at www.claude.ai or an API (Application Programming Interface) through Anthropic’s developer console. Below, we will explore the interesting data behind Claude AI including development trends, performance metrics, applications and user statistics. 

Development History

Since its inception, Claude has gone through a few upgrades and different iterations: 

• Claude 3.5 Haiku - October 22, 2024 (Source)
• Claude 1.0, anthropic’s first offering was released on March 14, 2023 (source)
• Claude 1.3- April 18, 2023 (source)
• Claude 2.0- July 11, 2023 (source)
• Claude 2.1- November 21, 2023 (source)
• Claude 3 (Opus & Sonnet) - March 4th, 2024 (Source)
• Claude 3 Haiku - March 13th, 2024 (Source)
• Claude 3.5 Sonnet - June 20th, 2024 (Source)
• Upgraded Claude 3.5 Sonnet - October 22, 2024 (Source)
• Claude 3.5 Haiku - October 22, 2024 (Source)

Anthropic has also released a streamlined and faster model with limited capabilities, called Claude Instant. This model has seen a similar development cycle as seen below: 

• Claude Instant 1.1- March 14, 2023 (source)
• Claude Instant 1.2- August 9, 2023 (source)

Technical Capabilities 

As mentioned earlier, within the past month, Anthropic has released an upgraded version of their leading AI model, Claude 2.1. Having also made advancements in their Claude Instant model within the last quarter, Anthropic has set its mark as a technical leader in the AI space. Below we can see different stats detailing the capabilities achieved by Claude AI:

• Claude 2 is trained on updated data from 2022 and early 2023 (source)
• Claude 2 can generate documents up to 4000 tokens (roughly 3000 words) (source)
• Claude AI (1.0) reads up to 75,000 words (source)
• Latest model, Claude 2.1, reads up to 150,000 words (source)
• Claude models are trained on over 137 billion text and code parameters (source)
• Claude allows file uploads of PDFs, DOCX, CSV, and TXT formats (source)
• Roughly 10% of Anthropic Claude’s dataset comes from non-English content (source)
• The Claude API is now available in 159 countries. (Source)
• One of Anthropic AI’s newest models, Claude 3 Opus, has 137 billion parameters, showcasing its prowess and exquisite handling in user queries and content generation (Source)
• Claude 3 uses data extraction technologies, advanced auto-completion, and can be easily integrated into live customer chats. (Source)
• Anthropic’s family of Claude 3 models continue to improve in speed. In their release, Anthropic notes that Claude Sonnet is twice as fast as Claude 2 and 2.1. (Source)

Performance Metrics 

In developing the different versions of Claude, the Anthropic team extensively tests and measures the performance of their advancing models. Below we can see some recent highlights in Claude AI performance: 

• The overall success of open-sourced LLMs in answering the 858 nephSAP multiple-choice questions correctly was 17.1% – 25.5%. In contrast, Claude 2 answered 54.4% of the questions correctly, whereas GPT-4 achieved a score of 73.3% (source)
• Claude 3 Opus achieved 86.8% accuracy on a MMLU general reasoning test – surpassing competitors such as Gemini and GPT-4. (Source)
• Claude Instant 1.2 scores 58.7% on the Codex evaluation, compared to 52.8% for Claude Instant 1.1. (source)
• It also achieves 86.7% on the GSM8K benchmark, compared to 80.9% for the previous version (source)
• Claude 2 also scored a 71.2% on the Codex HumanEval Python coding test (source)

Claude scored 88.0% on GSM8k grade-school math problems, showcasing its computational ability (source)

Aside from coding performance, Anthropic sees truthfulness, harmlessness, and helpfulness as pillars to Claude’s success. The following benchmarks were recently gathered by the Anthropic team in the wake of Claude 2.1’s release: 

• Claude AI shown to be 80% accurate in human evaluation using multi-turn question answer session (source)
• Claude 2.1 demonstrated a 30% reduction in incorrect answers when compared to claude 2.1 (source)

• Claude 2.1 also has a 3-4x lower rate of mistakenly concluding a document supports a particular claim (source)
• Claude 2.1 has also made significant gains in honesty, with a 2x decrease in false statements compared to the previous Claude 2.0 model. (source)
  ‍
• Claude 3 Haiku is the fastest AI model for its cost-effective pricing (Source).
• Claude 2.1 has also made significant gains in honesty, with a 2x decrease in false statements compared to the previous Claude 2.0 model. (source)
• Claude 3 Opus’s performance on standardized tests has an average LSAT of 161, an MBE of 85% and GRE in Quantitative, Verbal, and Writing tests respectively, at 159, 166 and 5.0 (2 shot). (Source)
• Claude 3 Opus has an average recall of 99.4% in all context lengths and 98.3% in 200k context length. (Source)
• Claude 3 Sonnet has an average of 95.4% in all context lengths and 91.4% in 200k context length. (Source)
• Claude 3 Haiku’s estimated context length is 95.9% for all lengths and 91.9% for 200k context lengths. (Source)

Source: www.anthropic.com

In more traditional benchmarks, the performance of Claude has been monitored while completing arduous standardized exams historically taken by humans. The following statistics measure the progress and improvements made by the different versions of Claude. 

Here’s a comparative overview of how Claude 1.3, Claude 2, Claude 3 Opus, Claude 3 Sonnet, and Claude 3 Haiku perform on standard exams (MBE - Law, GRE - Writing, HumanEval - Python, and GSM3K - Math):

• Claude 1.3:
  • Bar Exam (Law - MBE): 73% (Source)
  • GRE (Writing): N/A
  • HumanEval (Python): 56% (Source)
  • GSM8K (Math): 85.2% (Source)
• Claude 2:
  • Bar Exam (Law - MBE): 76.5% (Source)
  • GRE (Writing): 5.0 (Source)
  • HumanEval (Python): 71.2% (Source)
  • GSM8K (Math): 88% (Source)
• Claude 3 Opus:
  • Bar Exam (Law - MBE): 85% (Source)
  • GRE (Writing): 5.0 (Source)
  • HumanEval (Python): 84.9% (Source)
  • GSM8K (Math): 95.0% (Source)
• Claude 3 Sonnet:
  • Bar Exam (Law - MBE): 71% (Source)
  • GRE (Writing): N/A (Source)
  • HumanEval (Python): 73.0% (Source)
  • GSM8K (Math): 92.3% (Source)
• Claude 3 Haiku:
  • Bar Exam (Law - MBE): 64% (Source)
  • GRE (Writing): N/A (Source)
  • HumanEval (Python): 75.9% (Source)
  • GSM8K (Math): 88.9% (Source)

Note: The GRE writing test is scored from 0 to 6.0, the above graphic represents the available data on GRE writing scores as a percentage out of a total of 6.0.

Overview highlights:

• According to Anthropic, Claude 2 scored 76.5% in the Bar exam's multiple-choice section (GPT-3.5 achieved 50.3%)  (source)
• Claude 3 Opus has scored 85% on the BarExam (MBE), compared to GPT-4 with only 75.7% (Source)
• Claude also achieved a score higher than 90% of graduate school applicants in GRE reading and writing exams (source)
• Claude 3 Opus continues to score well on GRE writing exams with a 5.0 score. (source)

Applications & How to Access

Currently, the Claude AI models are available in about 159 countries worldwide (https://www.anthropic.com/claude-ai-locations). If unfortunately, you happen to reside outside those countries, it is still possible to use Claude through one of the methods below: 

• Slack- By using the enterprise version of the slack app, a Claude chatbot can be added to a workspace for free 

• Poe- Through Quora Poe, any user can sign up for a free account to access various LLM AI models including Claude Instant. Claude 2.0 is available to users who sign up for a paid premium account. 
• Vercel AI Playground- Similarly to Quora Poe, Vercel AI Playground allows users to access different LLM AI models. Vercel goes further to provide paid users the opportunity to use Claude and other LLM AI models simultaneously to compare outputs.  

Through the span of 2023, Anthropic has released its various models of Claude to the public. The following below are some quick notes on the current state of the accessibility of the Claude LLM AI models: 

• Anthropic offers Claude Pro at $20 per month (direct competitor to ChatGPT+) (source)
• Claude Pro allows up to 60,000 queries per month – 5X more than ChatGPT Plus (source)
• Claude Pro users can expect to send at least 100 messages to Claude 2 every eight hours (source)
• Claude does not link conversation logs with usernames, IP addresses, account info or other identifiers (source)
• Conversations remain anonymized (source)
• Complete conversations are temporarily stored to train Claude, but are deleted after 7 days. Only a small sample of conversations may be kept for up to 6 months for R&D (source)
• Claude Instant 1.2 is exclusively available as an API for businesses (source)
• According to our in house research, Claude AI is currently the least blocked LLM by top websites amongst its competitors like Google and OpenAI (source)
• Our study shows that as of December 2023, Claude AI is being blocked by 0.02% of the top 1000 websites blocking AI web crawlers (source)

• Claude 3’s premium options (Pro, Team, and Enterprise) include a variety of custom features such as early access to new features, audit logs, and data source integration (specific features vary by plan). (Source)
• Premium plans currently start at $18 per user per month for Pro, $25 for Team (based on annual subscriptions), and Enterprise pricing is available by contacting sales. (Source)

Originality.ai vs. Claude AI

New testing grounds have emerged with the recent developments of Anthropic’s Claude AI bot and the release of the Claude 3.5 Sonnet version. 

In these tests, we’ll observe how effective the Claude model is at concealing AI content by scanning the AI-generated articles with Originality.ai’s professional AI detection tool. The tests will include an analysis of how Originality.ai performs at detecting content when prompts include requests to “humanize” content.

While Claude can be exceptionally useful at providing ideas and generating valuable suggestions, creating entire articles with AI should be avoided, as Google can penalize AI content that doesn’t comply with spam policies.

For testing purposes, we will use the most recent version of Claude - Claude 3.5 Sonnet and the Originality.ai AI Checker. Now, let’s proceed with the first test and prompt Claude!

[Test 1] Common AI-Generated Claude 3.5 Sonnet Text

First, we’ll prompt Claude to generate a typical article without extra instructions (to create a baseline for comparison during future tests). Claude’s generative technology is similar to other chatbots, however, Anthropic has aimed to humanize Claude’s responses as much as possible.

Let’s begin with the first tests:

[Prompt #1] - Write a short article (500–1000 words) on the integration of artificial intelligence in 2024.

We’ve received a total of 693 words from Claude, covering the essentials of recent AI integration trends in 2024. Let’s check Originality.ai’s detection result:

Originality.ai detects the output from Claude as Likely AI with 100% Confidence.

Now, let’s attempt to humanize Claude 3.5 Sonnet’s output:

[Prompt #2] - Write a short article (500–1000 words) on the integration of artificial intelligence in 2024. Use a human tone and stick to the fluidity of a human conversation. Break up the text, include unique bullets, and implement numbered lists. Provide suggestions in first-person and try to use popular phrasings.

As a result of the second prompt, we’ve managed to extract an 863-word example from Claude. Let’s check Originality.ai’s detection result:

Even when prompted to ‘stick to the fluidity of a human conversation,’ Originality.ai continues to identify the content as AI-generated with 99% Confidence that the output is likely AI. 

The verdict from this round of testing? Providing Claude with extra instructions to create a more human-like tone in the prompt does not have a significant impact on the detection outcome.

Let’s proceed with the more complex tests, where we provide Claude with a human-written example of an article to use for comparison when generating text.

[Test 2] Generated Claude 3.5 Sonnet Text With Example

The unique capabilities of AI chatbots allow them to learn on the go via unique suggestions and user prompts. Let’s see what impact providing Claude with a unique article example has on AI detection.

We have provided Claude with a technical-themed example. Let’s have a look at the first prompt:

[Prompt #1] - Write a short article (500–1000 words) on the integration of artificial intelligence in 2024. Use this *article* as an example. Stick to the tone and structure of the provided article.

Similar to the first test, we won’t mention specific instructions that prompt it to humanize the content. The first prompt has generated a 622-word piece. Here are the detection results:

From this prompt, Originality.ai continues to identify the content as Likely AI. It detects the content as Likely AI with 75% Confidence (learn more about AI detection scores). The sections, which it detects as most likely generated by AI are highlighted in the deeper shades of red and orange.

Let’s move on with the second prompt and provide Claude with both an article example and instructions for content humanization:

[Prompt #2] - Write a short article (500–1000 words) on the integration of artificial intelligence in 2024. Use a human tone and stick to the fluidity of a human conversation. Break up the text, include unique bullets, and implement numbered lists. Provide suggestions in first-person and try to use popular phrasings. Use this *article* as an example. Stick to the tone and structure of the provided article.

Let’s compare the 692 words we’ve received with Originality.ai’s detection technology:

From this prompt, the AI Checker determined that the prompt was Likely AI with 100% Confidence, continuing to demonstrate that the detector identifies Claude’s content as AI-generated.

Overall Claude’s generated text was continuously identified as Likely AI by the Originality.ai AI detector. As new models are released, we’ll continue to evaluate the detectability of their text.

Pricing

Below is a pricing table comparing the costs of different Claude AI models as of May 2024 (source): 

• Claude Instant - Average price of $0.0008 per 1,000 input tokens and $0.0024 per 1,000 output tokens.

• Claude 2.0/2.1 - Average price of $0.008 per 1,000 input tokens and $0.024 per 1,000 output tokens.
• Claude 3 Opus - Average price of $0.015 per 1,000 input tokens and $0.075 per 1,000 output tokens.
• Claude 3 Sonnet - Average price of $0.003 per 1,000 input tokens and $0.015 per 1,000 output tokens.
• Claude 3 Haiku - Average price of $0.00025 per 1,000 input tokens and $0.00125  per 1,000 output tokens.
• Claude 3 Opus is currently available only for the West U.S.

Claude AI Traffic and Reach

As of September 2024, (latest available data), the Claude AI website has garnered widespread attention, reaching (source): 

• 65.9 million monthly visits 
• Increase in global ranking from 927th to 846th
• 31.45% bounce rate
• 3.74 pages per visit 
• 00:05:56 avg visit duration 
• 939th most visited site globally 
• 25.93% of the traffic is generated from the U.S.
• 24th most visited site in the Programming and Developer Software category

Claude AI shows great marketing potential as currently, the site reaches roughly 75.93% of its traffic through direct searches, the chart below shows how many users Claude AI reaches through other web traffic sources:

Similarly, this next graph illustrates the traffic driven to Claude AI by different social media channels (source):

• Youtube - 48.73%
• Whatsapp - 13.55%
• Facebook- 12.95% 
• Linkedin- 7.02%
• Instagram - 3.26% 
• Other - 14.48%

Claude AI User Demographics

 As of May 2024 (latest available data), the most common Claude AI can be described by the stats below (source): 

Gender: 

• Male - 64.79%
• Female - 35.21% 

Age: 

(Source)

• 18 - 24 =  23.31%
• 25 - 34 = 36.94%
• 35 - 44 = 18.20%
• 45 - 54 = 11.33%
• 55 - 64 = 6.37%
• 65+ = 3.85%

Geography (source): 

• United States - 25.93%
• India - 8.46%
• United Kingdom - 5.12%
• Korea - 3.36% 
• Japan - 3.35% 
• Rest of World - 53.79%

Summary

Time will tell whether Anthropic’s strategic decision to unveil Claude 2.1 during Sam Altman’s skirmish against OpenAI’s board of directors. Resounding market praise and support have made it clear that Anthropic is positioning itself as a frontrunner and key player in the AI field. By focusing on the HHH (honest, harmless, helpful) application of AI, Claude has found a niche in the market which is highlighted by the strength of the statistics listed above. With the continued improvements and advancements showcased by Anthropic and Claude AI, it is evident that AI as a field is at the onset of rapid transformation.