HuggingChat AI Detector Test – 10 Prompts Vs 4 AI Detectors
HuggingFace has launched a ChatGPT “clone” called HuggingChat based on Open Assistant a 30-billion-parameter LLaMa model. In this article, we do a quick study to see if any of the existing AI detectors on the market are capable of detecting Hugging Chat content. For a more in-depth AI detection accuracy study across multiple NLP models .
HuggingFace has launched a ChatGPT “clone” called HuggingChat based on Open Assistant a 30-billion-parameter LLaMa model. In this article, we do a quick study to see if any of the existing AI detectors on the market are capable of detecting Hugging Chat content.
To do a quick test we look at 10 prompts (including some of the most common AI detection “bypass” tricks), 2 human articles and then compare the results across 4 popular ChatGPT detectors to see how they measure up to HuggingChat content. This test was done all on video since clearly we have a bias and so to try and be as fair as possible the test is done on video so you can see everything including the prompt, writing the content and testing.
Finally, we do a deep dive into the pros and cons of HuggingChat vs ChatGPT for content creation.
First, let’s get to the results of the test…
Can HuggingChat Be Detected?
Yes, Originality.AI can detect HuggingChat, in this test 9/10 times. Other AI detectors struggled achieving only at best a 20% detection rate. Expect both HuggingChat to improve with new NLP models and AI detectors to improve at detecting Open Assistant powered HuggingChat.
Results from the test are shown below.
Green = Correctly Identified the Content as AI or Original
Yellow = Uncertain
Red = Incorrectly Identified the Content as AI or Original
Below is a screenshot of the HuggingChat for the test including the best that the web/reddit had to offer in terms of a prompt that can bypass AI detection.
Obviously, I am biased and so demonstrate a fair test I recorded a video of me doing performing this test.
The results clearly show that HuggingChat can be detected by Originality but fool most of the other AI detectors.
But don’t just believe this test go ahead and complete your own…
Go to https://huggingface.co/chat/
Test at AI Detectors (Originality.AI and any others you might want to use)
Here is a little background on HuggingChat and how it compares to ChatGPT.
In addition, it has created one of the original Hugging Face AI content detectors but it was created for GPT-2 and was reasonably accurate on GPT-3 but with ChatGPT and GPT-4 it is pretty useless.
Pros of HuggingChat vs ChatGPT
Free – HuggingChat is currently 100% free. Like OpenAI, Hugging Face itself is a company.
No Sign-in Required – Currently there is no sign-in required and you can simply go to the website and start interacting with HuggingChat.
Open Source – As AI continues to evolve it will be important that multiple organizations are harnessing the technology. If there were no alternatives then a world where OpenAI ONLY controlled the power of a tool like ChatGPT would not be ideal.
Privacy – HuggingChat does not store any chat data, use any data for training and there are no user accounts.
Ability to Evolve – With it being open source the number of ways it is going to be expanded is potentially more varied than the controlled variations of ChatGPT.
Cons of HuggingChat vs ChatGPT
Prompts require clearer instructions – Maybe it is because I have used it less but it felt like I needed to explain what I wanted as if it was a new employee compared to someone who has worked with me for years.
No Ability to “Continue” – Asking ChatGPT to create an article outline and then write that article requires you to type some version of “continue” when it reaches the character limit. ChatGPT seamlessly continues. HugginFace does not and goes off on a different tangent when you ask it to continue. For longer content, this will require some clever follow-up prompts and instructing HuggingChat to not generate too much content.
Decent but Not As Good as ChatGPT – Although not as good as ChatGPT (GPT-4) it is similar to ChatGPT when it initially launched.
Struggles with Reasoning – Similar to GPT-3 vs GPT-4 the HuggingChat model struggles with simple reasoning questions that GPT-4 is now capable of answering.
Overall I am impressed with this free open source alternative and it will be interesting to watch as it evolves.
The use of it as a free article writer seems very likely and so it will be critical that detection tools adapt. Currently Originality.AI is capable of detecting HuggingChat generated content but a larger study and then likely additional training will be required to have a detection accuracy similar to GPT-4 (99%).
Founder / CEO of Originality.AI I have been involved in the SEO and Content Marketing world for over a decade. My career started with a portfolio of content sites, recently I sold 2 content marketing agencies and I am the Co-Founder of MotionInvest.com, the leading place to buy and sell content websites. Through these experiences I understand what web publishers need when it comes to verifying content is original. I am not For or Against AI content, I think it has a place in everyones content strategy. However, I believe you as the publisher should be the one making the decision on when to use AI content. Our Originality checking tool has been built with serious web publishers in mind!