HuggingChat AI Detector Test – 10 Prompts Vs 4 AI Detectors

HuggingFace has launched a ChatGPT “clone” called HuggingChat based on Open Assistant a 30-billion-parameter LLaMa model. In this article, we do a quick study to see if any of the existing AI detectors on the market are capable of detecting Hugging Chat content.

For a more in-depth AI detection accuracy study across multiple NLP models checkout this article – AI Detection Accuracy

To do a quick test we look at 10 prompts (including some of the most common AI detection “bypass” tricks), 2 human articles and then compare the results across 4 popular ChatGPT detectors to see how they measure up to HuggingChat content. This test was done all on video since clearly we have a bias and so to try and be as fair as possible the test is done on video so you can see everything including the prompt, writing the content and testing.

Finally, we do a deep dive into the pros and cons of HuggingChat vs ChatGPT for content creation.

First, let’s get to the results of the test…

Can HuggingChat Be Detected?

Yes, Originality.AI can detect HuggingChat, in this test 9/10 times. Other AI detectors struggled achieving only at best a 20% detection rate. Expect both HuggingChat to improve with new NLP models and AI detectors to improve at detecting Open Assistant powered HuggingChat.

Results from the test are shown below.

• Green = Correctly Identified the Content as AI or Original
• Yellow = Uncertain
• Red = Incorrectly Identified the Content as AI or Original

Below is a screenshot of the HuggingChat for the test including the best that the web/reddit had to offer in terms of a prompt that can bypass AI detection.

Obviously, I am biased and so demonstrate a fair test I recorded a video of me doing performing this test.

The results clearly show that HuggingChat can be detected by Originality but fool most of the other AI detectors.

But don’t just believe this test go ahead and complete your own…

1. Go to https://huggingface.co/chat/
2. Create content
3. Test at AI Detectors (Originality.AI and any others you might want to use)

Here is a little background on HuggingChat and how it compares to ChatGPT.

What is HuggingFace?

Hugging Face is a community of AI open-source developers and companies. It has launched HuggingChat as an open-source competitor to OpenAI’s ChatGPT.

In addition, it has created one of the original Hugging Face AI content detectors but it was created for GPT-2 and was reasonably accurate on GPT-3 but with ChatGPT and GPT-4 it is pretty useless.

Pros of HuggingChat vs ChatGPT

• Free – HuggingChat is currently 100% free. Like OpenAI, Hugging Face itself is a company.
• No Sign-in Required – Currently there is no sign-in required and you can simply go to the website and start interacting with HuggingChat.
• Open Source – As AI continues to evolve it will be important that multiple organizations are harnessing the technology. If there were no alternatives then a world where OpenAI ONLY controlled the power of a tool like ChatGPT would not be ideal.
• Privacy – HuggingChat does not store any chat data, use any data for training and there are no user accounts.
• Ability to Evolve – With it being open source the number of ways it is going to be expanded is potentially more varied than the controlled variations of ChatGPT.

Cons of HuggingChat vs ChatGPT

• Prompts require clearer instructions – Maybe it is because I have used it less but it felt like I needed to explain what I wanted as if it was a new employee compared to someone who has worked with me for years.
• No Ability to “Continue” – Asking ChatGPT to create an article outline and then write that article requires you to type some version of “continue” when it reaches the character limit. ChatGPT seamlessly continues. HugginFace does not and goes off on a different tangent when you ask it to continue. For longer content, this will require some clever follow-up prompts and instructing HuggingChat to not generate too much content.
• Decent but Not As Good as ChatGPT – Although not as good as ChatGPT (GPT-4) it is similar to ChatGPT when it initially launched.
• Struggles with Reasoning – Similar to GPT-3 vs GPT-4 the HuggingChat model struggles with simple reasoning questions that GPT-4 is now capable of answering.

Conclusion

Overall I am impressed with this free open source alternative and it will be interesting to watch as it evolves.

The use of it as a free article writer seems very likely and so it will be critical that detection tools adapt. Currently Originality.AI is capable of detecting HuggingChat generated content but a larger study and then likely additional training will be required to have a detection accuracy similar to GPT-4 (99%).

‍