With the recent surge of GPTs (Generative Pre-Trained Transformers) and the marketplace store connecting developers and users, OpenAI has developed an ecosystem that allows developers to create tailored versions of ChatGPT to acutely meet the daily needs and workflow processes of its target consumers.
At Originality.ai, we are actively monitoring and studying the GPT market as well as the trends that lie beneath the numbers and will soon publish those insights. For now, we will look at the model behind the GPT store and custom GPTs, which also happens to be OpenAI’s most advanced publicly available LLM (Large Language Model), GPT-4.
Read below to dive further into the many different processes, statistics, and trends that have all converged to make GPT-4 possible.
What is GPT-4?
GPT-4, the latest milestone in OpenAI’s effort in scaling up deep learning. GPT-4 is a large multimodal model (accepting image and text inputs, emitting text outputs) that, while less capable than humans in many real-world scenarios, exhibits human-level performance on various professional and academic benchmarks. (Source)
When Was the Latest GPT-4 Model Announced/Released?
On Monday November 6, 2023 at the OpenAI DevDay event, company CEO Sam Altman announced a major update to its GPT-4 language model called GPT-4 Turbo, which can process a much larger amount of text than GPT-4 and features a knowledge cutoff of April 2023. (Source)
How can GPT-4 be Accessed?
GPT-4 currently sits behind a paywall. OpenAI has a subscription based model for consumers to access the more advanced forms of their ChatGPT model. Below are the current developments behind accessing GPT-4:
ChatGPT Plus subscribers will get GPT-4 access on chat.openai.com with a usage cap. (Source)
OpenAI may introduce a new subscription level for higher-volume GPT-4 usage; they also hope at some point to offer some amount of free GPT-4 queries so those without a subscription can try it too. (Source)
GPT-4 Architecture
A new report by SemiAnalysis reveals more details about OpenAI's GPT-4, concluding that "OpenAI is keeping the architecture of GPT-4 closed not because of some existential risk to humanity, but because what they've built is replicable”. (Source). As such, the following details stem from a recent GPT documentation leak and have not yet been confirmed by OpenAI:
GPT-4's Scale: GPT-4 has ~1.8 trillion parameters across 120 layers, which is over 10 times larger than GPT-3 (Source)
Mixture Of Experts (MoE): OpenAI utilizes 16 experts within their model, each with ~111B parameters for MLP. Two of these experts are routed per forward pass, which contributes to keeping costs manageable. (Source)
Dataset: GPT-4 is trained on ~13T tokens, including both text-based and code-based data, with some fine-tuning data from ScaleAI and internally. (Source)
Dataset Mixture: The training data included CommonCrawl & RefinedWeb, totaling 13T tokens. Speculation suggests additional sources like Twitter, Reddit, YouTube, and a large collection of textbooks. (Source)
Training Cost: The training costs for GPT-4 was around $63 million, taking into account the computational power required and the time of training. (Source)
Inference Cost: GPT-4 costs 3 times more than the 175B parameter Davinci, due to the larger clusters required and lower utilization rates. (Source)
Inference Architecture: The inference runs on a cluster of 128 GPUs, using 8-way tensor parallelism and 16-way pipeline parallelism. (Source)
Vision Multi-Modal: GPT-4 includes a vision encoder for autonomous agents to read web pages and transcribe images and videos. The architecture is similar to Flamingo. This adds more parameters on top and it is fine-tuned with another ~2 trillion tokens. (Source)
Does GPT-4 Really Utilize Over 100 Trillion Parameters?
When GPT-4 was first announced and subsequently released, it was heavily speculated that the new model was comprised of over 100 trillion parameters. After a couple months and a data leak containing some GPT-4 architecture details, the CEO of OpenAI, Sam Altman, was questioned about the matter:
When asked about one viral (and factually incorrect) chart that purportedly compares the number of parameters in GPT-3 (175 billion) to GPT-4 (100 trillion), Altman called it “complete bullshit.” (Source)
In reality, the reported parameter figure for GPT-4 is closer to 1 trillion. (Source)
Training Process
OpenAI spent 6 months iteratively aligning GPT-4 using lessons from their adversarial testing program as well as ChatGPT, resulting in their best-ever results (though far from perfect) on factuality, steerability, and refusing to go outside of guardrails. (Source)
To prepare the image input capability for wider availability, OpenAI began collaborating closely with a single partner to start. (Bemyeyes) (Source)
OpenAI makes the GPT codebase more publicly available by open-sourcing OpenAI Evals, their framework for automated evaluation of AI model performance, to allow anyone to report shortcomings in their models to help guide further improvements (Source)
Compared to GPT-3's 17 gigabytes of data, GPT-4, the most recent iteration of OpenAI, has 45 gigabytes of training data. As a result, GPT-4 can deliver significantly more accurate results than GPT-3. (Source)
What is GPT-4’s Context Length?
GPT-4 has a context length of 8,192 tokens. OpenAI is also slowly providing limited access to their 32,768–context (about 50 pages of text) version, gpt-4-32k, (Source)
Previously, GPT-4 featured an 8,000-token context window, with a 32K model available through an API for some developers; now this context window has been surpassed for turbo models. (Source)
That means GPT-4 Turbo can consider around 96,000 words in one go, which is longer than many novels. Also, a 128K context length can lead to much longer conversations without having the AI assistant lose its short-term memory of the topic at hand. (Source)
Introduction of GPT-4 Vision
Adding onto the text based capabilities of OpenAI’s GPT models, GPT-4 has introduced the possibility of interacting with GPT models through a visual capacity, look below to see the details behind “GPT-4-Vision”:
GPT-4 with vision (GPT-4V) enables users to instruct GPT-4 to analyze image inputs provided by the user. (Source)
Similar to GPT-4, training of GPT-4V was completed in 2022 and began providing early access to the system in March 2023 (Source)
GPT 4 Improvements on GPT-3.5
GPT-4 has proved to be a great success for OpenAI, making great improvements on the already impressive foundation that was established by ChatGPT and GPT-3.5. Below we can see some of the initial progress made by the new model and how it compares to the previous model, GPT-3.5:
In a casual conversation, the distinction between GPT-3.5 and GPT-4 can be subtle (Source)
GPT-4 is more reliable, creative, and able to handle much more nuanced instructions than GPT-3.5. (Source)
GPT-4 considerably outperforms existing large language models, alongside most state-of-the-art (SOTA) models which may include benchmark-specific crafting or additional training protocols (Source)
In the 24 of 26 languages tested, GPT-4 outperforms the English-language performance of GPT-3.5 and other LLMs (Chinchilla, PaLM), including for low-resource languages such as Latvian, Welsh, and Swahili (Source)
GPT-4 significantly reduces hallucinations relative to previous models (which have themselves been improving with each iteration). GPT-4 scores 40% higher than OpenAI’s GPT-3.5 on their internal adversarial factuality evaluations (Source)
OpenAI’s mitigations have significantly improved many of GPT-4’s safety properties compared to GPT-3.5. We’ve decreased the model’s tendency to respond to requests for disallowed content by 82% compared to GPT-3.5, and GPT-4 responds to sensitive requests (eg medical advice and self-harm) in accordance with their policies 29% more often. (Source)
And regarding cost, running GPT-4 Turbo as an API reportedly costs one-third less than GPT-4 for input tokens (at $0.01 per 1,000 tokens) and one-half less than GPT-4 for output tokens (at $0.03 per 1,000 tokens). (Source)
For many basic tasks, the difference between GPT-4 and GPT-3.5 models is not significant. However, in more complex reasoning situations, GPT-4 is much more capable than any of their previous models. (Source)
The following chart shows some of the progress made by each iteration of the GPT model when responding to legal inquiries:
GPT-4 API Pricing
With 128k context, fresher knowledge and the broadest set of capabilities, GPT-4 Turbo is more powerful than GPT-4 and offered at a lower price. (Source)
GPT-4 Turbo API Pricing
With broad general knowledge and domain expertise, GPT-4 can follow complex instructions in natural language and solve difficult problems with accuracy. (Source)
Comparing Latest GPT Models Available
As mentioned earlier, GPT-4 is a large multimodal model (accepting text or image inputs and outputting text) that can solve difficult problems with greater accuracy than any of the previous models, thanks to its broader general knowledge and advanced reasoning capabilities. Like gpt-3.5-turbo, GPT-4 is optimized for chat but works well for traditional completions tasks using the Chat Completions API. (Source)
GPT-4
GPT-4 Turbo
GPT-4 Areas for Improvement
Even though GPT-4 has made many strides in improving the performance of its preceding model, there still remains avenues for OpenAI to improve upon the model’s accuracy and reliability. As detailed below, GPT-4 still presents opportunities to improve when considering factualness, relevancy, and accuracy:
GPT-4 generally lacks knowledge of events that have occurred after the vast majority of its data cuts off (September 2021 and April 2023, depending on model version), and does not learn from its experience. (Source)
Despite its capabilities, GPT-4 has similar limitations as earlier GPT models. Most importantly, it still is not fully reliable (it “hallucinates” facts and makes reasoning errors). (Source)
When measuring legal acumen, GPT-4, like prior models, may still hallucinate sources, incorrectly interpret facts, or fail to follow ethical requirements; for the foreseeable future, the application should feature ”human-in-the-loop” workflows or similar safeguards (Source)
It should be noted that several of these topics where GPT-4 struggles are also areas where law students and bar examinees would also likely struggle. (Source)
The following metrics provided by OpenAI detail in-house testing that shows the gradual increases in accuracy scores for the different training methods used on their models. The scores reflect that although improvements have been made throughout the model’s generations, there is still much room for improvement:
Performance Metrics
Legal
On the MBE (Multistate Bar Examination), GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. (Source)
Contracts and Evidence are the topics with the largest overall improvement. GPT-4 achieves a nearly 40% increase over ChatGPT in Contracts and a more than 35% raw increase in Evidence. (Source)
Civil Procedure is both the worst subject for GPT-4, ChatGPT and human test-takers. However, Civil Procedure is a topic where GPT-4 was able to generate a 26% raw increase over ChatGPT. (Source)
Financial
Davinci and ChatGPT based on GPT-3.5 score 66% and 65% on the financial literacy test, respectively, compared to a baseline of 33%. However, ChatGPT based on GPT-4 achieves a near-perfect 99% score, pointing to financial literacy becoming an emergent ability of state-of-the-art models (Source)
GPT-4 obtained a near-perfect score of 99.3% (without the pre-prompt) and 97.4% (with a pre-prompt). Put differently, GPT-4 exhibits financial literacy: a basic, at the very least, grasp of financial matters. (Source)
The following table depicts the recent scores of GPT models when taking a financial literacy test. The models restrictions surrounding financial advice was circumvented by implementing the pre-prompt “You are a financial advisor”:
Current Commercial Uses of GPT-4
Be My Eyes, Visual Impairment Assistant
Beginning in March, 2023, Be My Eyes and OpenAI collaborated to develop Be My AI, a new tool to describe the visual world for people who are blind or have low vision. Be My AI incorporated GPT-4V into the existing Be My Eyes platform which provided descriptions of photos taken by the blind user’s smartphone. (Source)
Be My Eyes piloted ‘Be My AI’ from March to early August 2023 with a group of nearly 200 blind and low vision beta testers to hone the safety and user experience of the product. By September 2023, the beta test group had grown to 16,000 blind and low vision users requesting a daily average of 25,000 descriptions. (Source)
With the new visual input capability of GPT-4 (in research preview), Be My Eyes began developing a GPT-4 powered Virtual Volunteer™ within the Be My Eyes app that can generate the same level of context and understanding as a human volunteer. (Source)
The difference between GPT-4 and other language and machine learning models, explains Jesper Hvirring Henriksen, CTO of Be My Eyes, is both the ability to have a conversation and the greater degree of analytical prowess offered by the technology. (Source)
Duolingo, Language Learning
Duolingo turned to OpenAI’s GPT-4 to advance the product with two new features: Role Play, an AI conversation partner, and Explain my Answer, which breaks down the rules when you make a mistake, in a new subscription tier called Duolingo Max. (Source)
Duolingo engineers had tried using GPT-3 to supplement some of the human-powered features in its earlier chat feature. “It was close to being ready,” said lead engineer Bill Peterson, “but we didn’t feel it was at the point where we could confidently integrate it to handle the complex automated aspects of chats.” (Source)
GPT-4 has learned from enough public data to create a flexible back-and-forth for the learner.
With the new features, earners will be able to click “Explain my answer”, and GPT-4 will give an initial response. From there, the learner can return to the lesson, or get further explanation, and GPT-4 can dynamically update. (Source)
Icelandic Government, Language Preservation
With the help of private industry, Iceland has partnered with OpenAI to use GPT-4 in the preservation effort of the Icelandic language—and to turn a defensive position into an opportunity to innovate. (Source)
The partnership was envisioned not only as a way to boost GPT-4’s ability to service a new corner of the world, but also as a step towards creating resources that could serve to promote the preservation of other low-resource languages. (Source)
In a process called Reinforcement Learning from Human Feedback, or RLHF, human testers give GPT-4 a prompt, and four possible completions are generated. Testers then select the best answer from the four responses and edit it to create an ideal completion. The data from this process is then used to further train GPT-4 to produce better responses in the future. (Source)
Khan Academy, Education
In March 2023, Khan Academy announced that it will use GPT-4 to power Khanmigo, an AI-powered assistant that functions as both a virtual tutor for students and a classroom assistant for teachers. (Source)
The nonprofit began testing the newest version of OpenAI’s language model in 2022 and will initially make the Khanmigo pilot program available to a limited number of participants, though the public is invited to join the waitlist. (Source)
Adapting GPT-4 for teachers is also top of mind for Khan Academy. The nonprofit is testing out ways teachers could use GPT-4, such as writing classroom prompts or creating instructional materials for lessons. (Source)
Morgan Stanley, Wealth Management
Morgan Stanley wealth management deploys GPT-4 to organize its vast knowledge base. (Source)
Starting last year, the company began exploring how to harness its intellectual capital with GPT’s embeddings and retrieval capabilities—first GPT-3 and now GPT-4. (Source)
Morgan Stanley has trained GPT-4 to make the internal chatbot as helpful as possible for the company’s needs. Today, more than 200 employees are querying the system on a daily basis and providing feedback. (Source)
Stripe, Fraud Detection
Stripe leverages GPT-4 to streamline user experience and combat fraud. (Source)
Stripe had previously been using GPT-3 to help their support team better serve users through tasks like routing issue tickets and summarizing a user’s question. (Source)
Now, Stripe uses GPT-4 to scan these sites and deliver a summary, which outperforms those written by people. “When we started hand-checking the results, we realized, ‘Wait a minute, the humans were wrong and the model was right.’” Eugene Mann (Stripe Product Lead) says. “GPT-4 was basically better than human reviewers.” (Source)
Another critical way Stripe supports developers is through extensive technical documentation and a robust developer support team to answer technical questions or troubleshoot issues. GPT-4 is able to digest, understand and become that virtual assistant—almost instantly. (Source)
Just by analyzing the syntax of posts in Discord, GPT-4 has been flagging accounts where Stripe's fraud team should follow up and be sure it isn't, in fact, a fraudster playing nice. GPT-4 can help scan inbound communications, identifying coordinated activity from malicious actors. (Source)
Whoop, Fitness Assistant
After fine-tuning with anonymized member data and proprietary WHOOP algorithms, GPT-4 was able to deliver extremely personalized, relevant, and conversational responses based on a person’s data (Source)
Conclusion
Wrapping up, we can see by the following data and statistics how significant OpenAI’s latest advancement in their GPT technology has been. Not only has GPT-4 greatly improved upon the technical capabilities of its predecessors, it has also brought forth the creation of a new marketplace and platform for developers and creators to offer their own specialized and tailored GPT models to better assist and fill the personalized needs of consumers.
As detailed by the performance of GPT-4 in highly technical professional fields like law and finance, it is clear that we are on the horizon of an exciting technological revolution that will present endless opportunities to integrate GPT technology into industrial applications.
Moreover, with the partnerships OpenAI has negotiated to implement GPT commercially, we can also expect GPT-4 (and more advanced models) to make waves in other fields from education to entertainment. At Originality.ai, we are keen to continue monitoring the development of OpenAI’s GPT models to have a better understanding of the market dynamics behind GPTs.
Founder / CEO of Originality.AI I have been involved in the SEO and Content Marketing world for over a decade. My career started with a portfolio of content sites, recently I sold 2 content marketing agencies and I am the Co-Founder of MotionInvest.com, the leading place to buy and sell content websites. Through these experiences I understand what web publishers need when it comes to verifying content is original. I am not For or Against AI content, I think it has a place in everyones content strategy. However, I believe you as the publisher should be the one making the decision on when to use AI content. Our Originality checking tool has been built with serious web publishers in mind!
