Perplexity and Burstiness in Writing

When you stop and think about it, writing is actually pretty amazing. We’ve got this manifestation of thoughts and ideas that we put into a structured form of language so that others can understand us. But within that language itself, there are irregularities and unpredictabilities. Is there a way to account for these irregularities when it comes to machine learning and language models?

As it turns out, there is. Normally, when we discuss perplexity and burstiness in writing, we get down into the weeds of mathematical formulas and computational theories. Don’t worry though, I’m going to make this explanation as painless as possible so that you can be a better writer without having to crunch all the numbers or analyze your content in painstaking detail.

How We Use Words and How Often We Use Them

Perplexity and burstiness essentially breaks down to: how we use words and how often we use them. As writers and content creators, sometimes our writing is predictable. Beyond that, sometimes the words we use pop up suddenly in groups or “bursts”.  Let’s break down each one in turn. 

What is Perplexity?

The mathematical formula for calculating perplexity looks like this:

‍

Essentially this formula calculates how predictable the next word is. Imagine reading a book and trying to guess what the next word will be. If you can guess it without batting an eye, that’s a low level of perplexity. If it’s more difficult to guess, that’s high perplexity. With this formula, the lower the number, the easier it is to guess the words.

So what does that mean for you as a writer? If your writing is too predictable, it might come across as boring. If it’s too unpredictable, it can be confusing and hard to understand. Having a good balance is crucial to engage your readers and communicate clearly and concisely. 

What is Burstiness? 

At times, we writers can get pretty comfortable in our use of words, to the point where we use some words again and again. Imagine you’re writing about a birthday party. In describing it, you’ll mention the word “cake” about a dozen times, but after describing the scene, you probably won’t talk about the cake again much at all. That’s burstiness. 

The same happens in fiction when introducing characters. You’ll mention their names often at the beginning, but less so toward the end, because people already understand who they are. That’s burstiness in a nutshell. 

Why does that matter to you as a content professional? It means that using words in bursts can change the focus of your story or narrative. Some writers employ this often as a matter of style. At the same time, repeating words can make your point stronger. As with perplexity, finding a good balance is crucial to the quality of your writing. 

How Do Perplexity and Burstiness Interact?

They may seem like two totally separate concepts but both perplexity and burstiness overlap. If a text has lots of bursts, it may be harder to guess the next word because the text is overall less uniform. Some types of writing or articles may have more bursts than others. And much of it is up to the individual writer’s personal style. 

As AI continues to develop and tools become more attuned to the nuances and subtleties of human language (including perplexity and burstiness), you can expect these platforms to offer more than just grammar and spelling suggestions. In the very near future, these tools might be able to tell writers if they’re being too predictable or suggest when to use more burstiness in their language. As a result, writers can make their content more engaging, mix up their style and play around with the predictability of their writing. 

What Does This Mean for Your Writing? 

Burstiness plays a role in nearly every type of content. Imagine sprinkling in terms related to warfare in a romance novel. That will indicate to the reader that there’s a battle or conflict about to happen. Some genres may use burstiness more than others. Scientific and academic texts in particular frequently use specific terms in sections which create bursty patterns. 

In other cases, burstiness is deliberate. Martin Luther King Jr.’s famous “I Have a Dream” speech uses repetition effectively to make his points more emphatic and memorable. In your own writing, regulating the levels of burstiness makes your content more engaging and helps to keep your readers’ attention for longer periods of time. 

What Do Perplexity and Burstiness Have to Do With AI Training? 

So why talk about mathematical formulas and predicting words or identifying clusters of words at all? Why does it matter? Because this very same information is used to train AI tools, as well as AI detection tools like Originality.ai. When an AI system is trained to understand and generate language, it learns from vast amounts of text while trying to predict what word comes next based on the words it has seen so far.  At the core, it’s all about recognizing patterns. 

Perplexity measures how well AI can predict the next word. If it guesses correctly most of the time, it has low perplexity. If it is often wrong, the perplexity is high. Understandably, for AI to work as well as it does, developers want to try out different models and evaluate different processes to choose the one that works best. Perplexity is their way of comparing – models with lower perplexity are generally better. 

And what about burstiness? Since burstiness is a natural and inherent part of human language, AI tools need to be able to recognize and reproduce this pattern. If an AI is fed fiction novels, it needs to know that if a new character is introduced, their name will come up several times in a short period of time. 

At the same time, we don’t want the AI to overuse words or get stuck in a repetitive loop. We want the writing to sound as natural as possible, where possible. This means that AI developers have to find a balance, just as human writers do. It also means that developers need to train the AI on different types of text, not just one specific genre or one specific type of content. Doing this teaches the AI different patterns. 

Why Does All of This Matter? 

There are two main reasons why we go to such lengths to teach and train the AI (and conversely, AI detection systems): 

The first is that developers want AI to be able to communicate effectively with humans. In order to do that, it needs to sound as natural as possible, and in order to sound natural, developers have to understand and adjust the levels of perplexity and burstiness. The end result helps the AI to generate text that’s similar to what a human would say. 

The second is that as more data is gathered on how well the AI is performing, engineers and developers can train it to perform better in a subsequent version. Finding the right levels of perplexity and burstiness isn’t a “one size fits all” solution, which means that AI tools are consistently being developed and refined to better match how humans write and speak. 

To sum it up, being able to understand perplexity and burstiness is important not only as a writer, but also as an AI development and detection service. Tweaking AI and machine learning formulas so that it can understand and reproduce human-like patterns in speech is a core step to making AI-generated content more authentic and engaging (while helping AI detection tools in turn become equally more refined and adept at uncovering it).