The generative large language model (LLM) space is constantly evolving with new innovations. Renowned companies like OpenAI, Google, Meta, Nvidia, and Microsoft are all competing to build the best foundational LLM models.
At the same time, new startups are emerging, creating new models or fine-tuning existing ones. So, they’re keen to understand how they stack up — who's leading, innovating, and growing? However, establishing an industry benchmark for LLM models can be challenging.
We used the Elo rating system from Large Model Systems Organization (LMSYS) to rank 35 organizations based on their top model performance from the Chatbot Arena.
We also collected data on:
- Number of models in the Chatbot Arena
- A list of models
- Market valuations
- Number of employees
- Monthly visits
- Total funding amount for private companies
Then, we added the organization's website as a comparison and performance reference.
The data is presented in an Airtable, displaying organization rankings and top metrics. Click on each organization to explore the metrics in detail.
Note: This table will be updated regularly as the Chatbot Arena model’s rankings and the organization’s standings change.
Industry Benchmarks and Large Model Systems Organization (LMSYS)
Large Model Systems Organization (LMSYS Org) is an open research organization founded by students and faculty from UC Berkeley in collaboration with UCSD and CMU. They adopted a pairwise comparison ranking system known as the Elo rating system, also officially used by the US Chess Federation to rank chess players.
The LMSYS Chatbot Arena Leaderboard rates and ranks LLM models.
- Over 1,000,000 human pairwise comparisons were made to rank LLMs using the Bradley-Terry model.
- The ratings are displayed with the Elo-scale.
The system is gaining traction in the industry by providing a transparent, competitive framework for model evaluation.
Note: For the latest information on the ranking system that LMSYS uses, visit their blog post, ‘Chatbot Arena: New models & Elo system update.’
Features Overview
A quick overview of some of the features that are included in our table.
Rank
Rankings are determined by the Top Model ELO score. The organization with the highest Top Model ELO is ranked first, and so on. This provides a clear view of which organizations have leading models based on the Elo rating system.
Top Model ELO
This score represents the Elo rating of the highest-rated model for each organization. Elo ratings are a method for calculating the relative skill levels of players (or models, in this case) in competitor-versus-competitor games. The higher the Elo score, the better the model is considered to be.
Monthly Visits
Sourced from SEMRush via Crunchbase, this metric indicates the number of visits the organization’s website receives monthly to provide insight into the organization’s online presence and reach.
Model Count
This is the total number of models an organization has in the Chatbot Arena. It gives an idea of the organization’s presence and diversity in the model landscape.
Votes
This column aggregates the total number of pairwise comparison votes received by all models of an organization in the Chatbot Arena. Each vote represents a user’s choice, compared between two models, contributing to the Elo rating.
Total Funding Amount
This column shows the total amount of funding an organization has raised, sourced from Crunchbase. It reflects the financial backing and investor confidence in the organization's projects and potential.
Valuation
This column provides the estimated market value of the organization. For public companies, this data is straightforward, while for private companies, it is often estimated based on funding rounds and press release news.
Number of Employees (string)
This data comes from Crunchbase and is typically presented as a range, such as ‘251-500.’ It’s an approximation because the data is aggregated from various sources and cross-referenced to provide a close estimate rather than an exact number like those reported by public companies in their quarterly filings.
Last Updated
This feature indicates the last time the Airtable data was updated.
Top 10 Organizations Overview
1. OpenAI
- Top Model ELO: 1,287
- Models: GPT-4o-2024-05-13, GPT-4-Turbo-2024-04-09, GPT-4-1106-preview, GPT-4-0125-preview, GPT-4-0314, GPT-4-0613, GPT-3.5-Turbo-0613, GPT-3.5-Turbo-0314, GPT-3.5-Turbo-0125, GPT-3.5-Turbo-1106
- Model Count: 10
- Total Funding Amount: $11,300,000,000
- Valuation: $80,000,000,000
- Monthly Visits: 2,490,602,320
- Description: OpenAI is an AI research and deployment company that conducts research and implements machine learning.
- Website: openai.com
2. Anthropic
- Top Model ELO: 1272
- Models: Claude 3.5 Sonnet, Claude 3 Opus, Claude 3 Sonnet, Claude 3 Haiku, Claude-1, Claude-2.0, Claude-2.1, Claude-Instant-1
- Model Count: 7
- Total Funding Amount: $7,559,000,000
- Valuation: $18,400,000,000
- Monthly Visits: 10,050,430
- Description: Anthropic is an AI safety and research company that focuses on increasing the safety of large-scale AI systems.
- Website: anthropic.com
3. Google
- Top Model ELO: 1,267
- Models: Gemini-Advanced-0514, Gemini-1.5-Pro-API-0514, Gemini-1.5-Pro-API-0409-Preview, Gemini-1.5-Flash-API-0514, Bard (Gemini Pro), Gemini Pro (Dev API), Gemini Pro, Gemma-1.1-7B-it, Gemma-7B-it, Gemma-1.1-2B-it, PaLM-Chat-Bison-001, Gemma-2B-it
- Model Count: 12
- Total Funding Amount: $26,000,000
- Valuation: $2,170,033,000,000
- Monthly Visits: 164,568,726,251
- Description: Google is a multinational corporation that specializes in Internet-related services and products.
- Website: google.com
4. 01 AI
- Top Model ELO: 1,241
- Models: Yi-Large-preview, Yi-Large, Yi-1.5-34B-Chat, Yi-34B-Chat
- Model Count: 4
- Total Funding Amount: $10,050,430
- Valuation: $1,000,000,000
- Monthly Visits: 5,241
- Description: 01.AI is developing a language model for the Chinese market that enhances productivity, creating significant economic and societal value.
- Website: 01.ai
5. Nvidia
- Top Model ELO: 1,208
- Models: Nemotron-4-340B-Instruct, NV-Llama2-70B-SteerLM-Chat
- Model Count: 2
- Total Funding Amount: $4,095,000,000
- Valuation: $3,335,268,000,000
- Monthly Visits: 72,292,416
- Description: NVIDIA is a computing platform operating at the intersection of graphics, HPC, and AI.
- Website: nvidia.com
6. Meta
- Top Model ELO: 1,207
- Models: Llama-3-70b-Instruct, Llama-3-8b-Instruct, Llama-2-70b-chat, Llama-2-13b-chat, CodeLlama-70B-instruct, CodeLlama-34B-instruct, Llama-2-7b-chat, LLaMA-13B
- Model Count: 8
- Total Funding Amount: $24,607,000,000
- Valuation: $1,267,661,000,000
- Monthly Visits: 17,890,429,543
- Description: Meta is a social technology company that enables people to connect, find communities, and grow businesses.
- Website: meta.com
7. Zhipu AI
- Top Model ELO: 1207
- Models: GLM-4-0520, GLM-4-0116
- Model Count: 2
- Total Funding Amount: $734,000,000
- Valuation: $1,000,000,000
- Monthly Visits: 17,333
- Description: Zhipu AI is a data and knowledge artificial intelligence company.
- Website: zhipuai.cn
8. Reka AI
- Top Model ELO: 1,200
- Models: Reka-Core-20240501, Reka-Flash-Preview-20240611, Reka-Flash-21B-online, Reka-Flash-21B
- Model Count: 4
- Total Funding Amount: N/A
- Valuation: $1,000,000,000
- Monthly Visits:
- Description: Reka AI is a globally distributed foundation model startup headquartered in Sunnyvale, California. It builds multimodal artificial intelligence to empower organizations and businesses.
- Website: reka.ai
9. Cohere
- Top Model ELO: 1,190
- Models: Command R+, Command R
- Model Count: 2
- Total Funding Amount: N/A
- Valuation: $885,000,000
- Monthly Visits: 400,214
- Description: Cohere provides access to advanced Large Language Models and NLP tools through one easy-to-use API.
- Website: cohere.com
10. Alibaba
- Top Model ELO: 1,188
- Models: Qwen2-72B-Instruct, Qwen-Max-0428, Qwen1.5-110B-Chat, Qwen1.5-72B-Chat, Qwen1.5-32B-Chat, Qwen1.5-14B-Chat, Qwen1.5-7B-Chat, Qwen-14B-Chat, Qwen1.5-4B-Chat
- Model Count: 9
- Total Funding Amount: $13,907,000,000
- Valuation: $168,432,000,000
- Monthly Visits: 124,679,060
- Description: Alibaba Group enables businesses to transform the way they market, sell, and operate to improve their efficiency.
- Website: alibabagroup.com
Data Sources
Chatbot Arena
In brief, Chatbot Arena by LMSYS Org is an open research platform from UC Berkeley in collaboration with UCSD and CMU. It uses the Elo rating system to rank LLMs through over 1,000,000 human pairwise comparisons. It’s a transparent and competitive framework for model evaluation, making it an emerging benchmark in the industry.
Link: Chatbot Arena
Crunchbase
Crunchbase provides comprehensive data on private companies, including financial information, the number of employees, and web traffic. This data helps contextualize the business environment and operational scale of each organization in our ranking.
Link: Crunchbase
Methodology
To compile the rankings and data for the top organizations developing LLMs, we followed these steps.
Step 1: Data Collection
- Extracted Elo ratings for models from the Chatbot Arena leaderboard.
- Gathered organizational data (e.g., number of employees, funding amount, monthly visits) from Crunchbase.
Step 2: Data Aggregation
- Ranked organizations based on their top model's Elo rating.
- Aggregated additional models under each organization to provide a comprehensive view and count of their contributions to the LLM space.
Step 3: Manual Verification
- Conducted web searches to verify and supplement valuation data for private companies not fully covered by Crunchbase.
Step 4: Data Presentation
- Presented data in an Airtable card layout, enabling an intuitive and interactive user experience. The aim was to make this dashboard a quick and easy resource to keep track of the top organizations in the generative LLM space.
