Thread by @TivadarDanka

There are two mistakes people make when building predictive models.

Most models are either too simple to be useful or too complicated to be used. Learning to pick just the right one is a valuable skill that can be applied anywhere.

Here is the basic principle behind it. ↓

Let's see some movie review sentiment analysis to illustrate the point!

Here is a short review snippet: "I had the terrible misfortune of having to view this b-movie in its entirety."

Without a doubt, this review is negative.

Based on this sample, one possible way to capture the sentiment could be just simply looking for the word "horrible".

Reviews containing it are predicted as negative. Otherwise, they are positive.

Now take a look at this review.

"It was fun to watch. But yea, definitely not a bad/horrible movie."

Our model predicts this as negative, despite being a positive review.

What happened? The model failed to capture contextual information, leaving us with false predictions.

Now, let's look at the opposite end of the spectrum.

With its 175 billion parameters, the GPT-3 model surely has the capacity to capture the relation between text and its sentiment.

However, would that be feasible? Not really.

GPT-3 is so huge that its training costs millions of USD, takes weeks, and even the parameters are stored in several distinct computers.

Using a large model for prediction is difficult as well. It may run on servers, but definitely not, for example, on a mobile phone.

Thus, a hyper-accurate and complex model can fail just as much as an oversimplified one, but in a different way.

We need to pick one that is complex enough to be accurate, but simple enough to be useful.

This zone in the middle, is called the Goldilocks zone.

If you would like to visualize this, here is an excellent illustration and explanation.

https://twitter.com/alaingoriely/status/1090577668888776707

The term "Goldilocks zone" comes from the children's story The Three Bears, where a young girl named Goldilocks has to pick between three different bowls of food.

She chooses the one that is not too hot or not too cold, but just the right temperature.

This principle is so universal that our world is full with its applications.

For instance, in astrobiology, the habitable zone around a star is called the Goldilocks zone.

To sum up, always include the necessary variables in your model, but not more. This applies to other areas of life: going all the way can be too much, but putting at least some energy is required.

Just like sports. I am not an athlete, yet I do sports every day for my health.