Statistics

75+ Claude AI Model Statistics in Q4 2023

Explore the latest in AI advancements with Anthropic's Claude 2.1, a HHH AI assistant. Discover trends, performance metrics, applications, and user stats.

Jonathan Gillham

In November 2023, the AI, and at a larger scale, the tech world was feverishly consumed by news of an incipient coup staged by the Open AI (the organization behind ChatGPT) board of directors. Ultimately, that coup failed to result in executive reform. Still, it did provide the perfect opportunity for one of its biggest competitors, Anthropic, to draw attention to its own LLM (large language model). As a result, on Tuesday, Nov 21st, 2023,  Anthropic announced grand updates to its AI model, Claude. Considering the enthusiastic market response, we will look at the many developing trends and statistics behind Claude AI. 

Claude, now in its 2.1 version, is an AI (artificial intelligence) based assistant developed by Anthropic using their research into HHH (Helpful, Honest, Harmless) AI applications. The assistant can be accessed through a chatbot at www.claude.ai or an API (Application Programming Interface) through Anthropic’s developer console. Below, we will explore the interesting data behind Claude AI including development trends, performance metrics, applications and user statistics. 

Development History

Since its inception, Claude has gone through a few upgrades and different iterations: 

  • Claude 1.0, anthropic’s first offering was released on March 14, 2023 (source)
  • Claude 1.3- April 18, 2023 (source)
  • Claude 2.0- July 11, 2023 (source)
  • Claude 2.1- November 21, 2023 (source)

Anthropic has also released a streamlined and faster model with limited capabilities, called Claude Instant. This model has seen a similar development cycle as seen below: 

  • Claude Instant 1.1- March 14, 2023 (source)
  • Claude Instant 1.2- August 9, 2023 (source)

Technical Capabilities 

As mentioned earlier, within the past month, Anthropic has released an upgraded version of their leading AI model, Claude 2.1. Having also made advancements in their Claude Instant model within the last quarter, Anthropic has set its mark as a technical leader in the AI space. Below we can see different stats detailing the capabilities achieved by Claude AI:

  • Claude 2 is trained on updated data from 2022 and early 2023 (source)
  • Claude 2 can generate documents up to 4000 tokens (roughly 3000 words) (source)
  • Claude AI (1.0) reads up to 75,000 words (source)
  • Latest model, Claude 2.1, reads up to 150,000 words (source)
  • Claude models are trained on over 137 billion text and code parameters (source)
  • Claude allows file uploads of PDFs, DOCX, CSV, and TXT formats (source)

Roughly 10% of Anthropic Claude’s dataset comes from non-English content (source)

Claude AI Technical Capabilities

Performance Metrics 

In developing the different versions of Claude, the Anthropic team extensively tests and measures the performance of their advancing models. Below we can see some recent highlights in Claude AI performance: 

  • The overall success of open-sourced LLMs in answering the 858 nephSAP multiple-choice questions correctly was 17.1% – 25.5%. In contrast, Claude 2 answered 54.4% of the questions correctly, whereas GPT-4 achieved a score of 73.3% (source)
  • Claude Instant 1.2 scores 58.7% on the Codex evaluation, compared to 52.8% for Claude Instant 1.1. (source)
  • It also achieves 86.7% on the GSM8K benchmark, compared to 80.9% for the previous version (source)
  • Claude 2 also scored a 71.2% on the Codex HumanEval Python coding test (source)

Claude scored 88.0% on GSM8k grade-school math problems, showcasing its computational ability (source)

Claude AI Coding Performance

Aside from coding performance, Anthropic sees truthfulness, harmlessness, and helpfulness as pillars to Claude’s success. The following benchmarks were recently gathered by the Anthropic team in the wake of Claude 2.1’s release: 

  • Claude AI shown to be 80% accurate in human evaluation using multi-turn question answer session (source)
  • Claude 2.1 demonstrated a 30% reduction in incorrect answers when compared to claude 2.1 (source)
  • Claude 2.1 also has a 3-4x lower rate of mistakenly concluding a document supports a particular claim (source)

Claude 2.1 has also made significant gains in honesty, with a 2x decrease in false statements compared to the previous Claude 2.0 model. (source)

Human Feedback Evaluations

Source: www.anthropic.com

In more traditional benchmarks, the performance of Claude has been monitored while completing arduous standardized exams historically taken by humans. The following statistics measure the progress and improvements made by the different versions of Claude: 

  • According to Anthropic, Claude 2 scored 76.5% in the Bar exam's multiple-choice section (GPT-3.5 achieved 50.3%)  (source)

Claude also achieved a score higher than 90% of graduate school applicants in GRE reading and writing exams (source)

Claude Standardized Exam Performance

Applications & How to Access

Currently, the Claude AI models are available in about 95 countries worldwide (https://www.anthropic.com/claude-ai-locations). If unfortunately, you happen to reside outside those countries, it is still possible to use Claude through one of the methods below: 

  • Slack- By using the enterprise version of the slack app, a Claude chatbot can be added to a workspace for free 
  • Poe- Through Quora Poe, any user can sign up for a free account to access various LLM AI models including Claude Instant. Claude 2.0 is available to users who sign up for a paid premium account. 
  • Vercel AI Playground- Similarly to Quora Poe, Vercel AI Playground allows users to access different LLM AI models. Vercel goes further to provide paid users the opportunity to use Claude and other LLM AI models simultaneously to compare outputs.  

Through the span of 2023, Anthropic has released its various models of Claude to the public. The following below are some quick notes on the current state of the accessibility of the Claude LLM AI models: 

  • More than 350,000 people signed up to the waitlist to use Claude 2, the most recent version of the language model. (source)
  • Anthropic offers Claude Pro at $20 per month (direct competitor to ChatGPT+) (source)
  • Claude Pro allows up to 60,000 queries per month – 5X more than ChatGPT Plus (source)
  • Claude Pro users can expect to send at least 100 messages to Claude 2 every eight hours (source)
  • Claude does not link conversation logs with usernames, IP addresses, account info or other identifiers (source)
  • Conversations remain anonymized (source)
  • Complete conversations are temporarily stored to train Claude, but are deleted after 7 days. Only a small sample of conversations may be kept for up to 6 months for R&D (source)
  • Claude Instant 1.2 is exclusively available as an API for businesses (source)
  • According to our in house research, Claude AI is currently the least blocked LLM by top websites amongst its competitors like Google and OpenAI (source)
  • Our study shows that as of December 2023, Claude AI is being blocked by 0.02% of the top 1000 websites blocking AI web crawlers (source)

Pricing

Below is a pricing table comparing the costs of different Claude AI models as of November 2023(source): 

  • Claude Instant Avg (Prompt and Completion): $0.0714USD per 7500 words
  • Claude 2.0 & 2.1 Avg (Prompt and Completion): $0.16USD per 7500 words
Claude AI Model Pricing

Claude AI Traffic and Reach

As of November, 2023 (latest available data), the Claude AI website has garnered widespread attention, reaching (source): 

  • 25.9 million monthly visits 
  • 5.2% increase in visits compared to October 2023 
  • 42.87% bounce rate 
  • 3.31 pages per visit 
  • 00:04:38 avg visit duration 
  • 2,835th most visited site globally 
  • 1,907th most visited site in the United States 
  • 51st most visited site in the Programming and Developer Software category 

Claude AI shows great marketing potential as currently, the site reaches roughly 21.68 million users through direct searches, the chart below shows how many users Claude AI reaches through other web traffic sources:

Claude AI Traffic By Source

Similarly, this next graph illustrates the traffic driven to Claude AI by different social media channels (source):

  • Youtube- 53.34%
  • Facebook- 10.9% 
  • Twitter/ X- 9.77% 
  • Linkedin- 9.68%
  • Whatsapp Messenger- 6.1% 
  • Other- 10.21%
Social Media Traffic to Claude AI

Claude AI User Demographics

 As of October 2023 (latest available data), the most common Claude AI can be described by the stats below (source): 

Gender: 

  • Male- 64.4%
  • Female- 36.6%
Claude AI Users By Gender

Age: 

  • 18 - 24 =  24.79%
  • 25 - 34 = 36.13%
  • 35 - 44 = 17.94%
  • 45 - 54 = 11.53%
  • 55 - 64 = 5.99%
  • 65+ = 3.62%
Claude AI Users By Age

Geography (source): 

  • United States- 44.92%
  • United Kingdom- 8.45%
  • India- 4.21%
  • China- 3.09% 
  • Taiwan- 3.08% 
  • Rest of World- 36.25%
Claude AI Users By Location

Summary

Time will tell whether Anthropic’s strategic decision to unveil Claude 2.1 during Sam Altman’s skirmish against OpenAI’s board of directors. Resounding market praise and support have made it clear that Anthropic is positioning itself as a frontrunner and key player in the AI field. By focusing on the HHH (honest, harmless, helpful) application of AI, Claude has found a niche in the market which is highlighted by the strength of the statistics listed above. With the continued improvements and advancements showcased by Anthropic and Claude AI, it is evident that AI as a field is at the onset of rapid transformation.

Jonathan Gillham

Founder / CEO of Originality.AI I have been involved in the SEO and Content Marketing world for over a decade. My career started with a portfolio of content sites, recently I sold 2 content marketing agencies and I am the Co-Founder of MotionInvest.com, the leading place to buy and sell content websites. Through these experiences I understand what web publishers need when it comes to verifying content is original. I am not For or Against AI content, I think it has a place in everyones content strategy. However, I believe you as the publisher should be the one making the decision on when to use AI content. Our Originality checking tool has been built with serious web publishers in mind!

More From The Blog

AI Content Detector & Plagiarism Checker for Serious Content Publishers

Improve your content quality by accurately detecting duplicate content and artificially generated text.