Thread by @svpino

A topic that comes up in every interview.

Bias, variance, and their relationship with machine learning algorithms. One of the most basic concepts that you have to know by heart.

Here is a simple summary that you will easily remember.

↓

Every machine learning algorithm deals with 3 types of errors:

1. Bias error
2. Variance error
3. Irreducible error

There's nothing we can do about #3.

Let's focus on the other two.

↓ 1/5

"Bias" refers to the assumptions the model makes to simplify the process of finding answers.

The more assumptions it makes, the more biased the model is.

"Variance" refers to how much the answers given by the model will change if we use different training data.

If the answers stay the same regardless of the data, the model has low variance.

Often, linear models are high-bias, and nonlinear models are low-bias.

Example low-bias algorithms:
• Decision Trees
• SVN
• kNN

Example high-bias algorithms:
• Linear Regression
• Logistic Regression

Often, linear models are low-variance, and nonlinear models are high-variance.

Example low-variance algorithms:
• Linear Regression
• Logistic Regression

Example high-variance algorithms:
• Decision Trees
• SVN
• kNN

Sometimes, you can change how these algorithms work to get a different tradeoff between their bias and variance.

Example:

• By increasing the value of "k" in kNN, we can increase the algorithm's bias.

• By pruning a Decision Tree, we can reduce its variance.

It doesn't matter what you do; the tradeoff is always there:

• Increasing bias decreases variance.
• Increasing variance decreases bias.

To work around this:

• Choose the appropriate algorithm
• Configure it correctly
• Work with the underlying dataset

If you want low-bias and low-variance machine learning content, follow me @svpino.

I come here to write about machine learning, and I promise you'll enjoy it.