Thread by @svpino

I built a machine learning model that predicts whether your car will crash today.

And it's 99% accurate!

Here is the secret: ↓

This thread is the answer to this question.

Before getting into the details, let's jump right into the source code of my model:

The National Safety Council reports that your odds of being in a car crash in the United States are less than 1%.

This means that even my dumb function that always returns False is very accurate!

The problem here is probably obvious by now:

Accuracy is not a good metric when you face a very imbalanced problem.

You can achieve very high accuracy even with a model that does absolutely nothing useful.

Some examples of imbalanced problems:

• Detecting fraudulent transactions
• Classifying spam messages
• Determining if a patient has cancer

Depending on the problem and what do you care about, the following three metrics will give you much more information than the accuracy:

• Recall
• Precision
• F1-Score

As an example, recall is defined as follows: TP / (TP + FN)

For our car crash problem, the recall of the dummy solution is 0 (since we never detect positive examples.)

Clearly, that solution is garbage.

Getting back to the original question, Option 4 is the answer.

All the other three choices are helpful for imbalanced problems.

By the way, I write practical tips, break down complex concepts, and regularly publish short quizzes to keep you on your toes.

Follow me @svpino, and let's do this together!