Thread by @ammaryh92

The ABSOLUTE ESSENTIALS of Bias/Variance Analysis

🧵This thread will cover the following concepts:
a. Bayes Error
b. Bias vs Variance
c. Possible Solutions

(Explanation + Examples)

#MachineLearning #DataScience

📜Introduction
- After training a ML model, it is important to assess its performance before putting it into production.
- We start by measuring the model performance on the training set to evaluate how well the model fits the training data.

- Then we measure the model performance on the test set to evaluate the generalization error.

To measure the model performance on the training set, we need a reference value against which we can compare the model performance.
This reference value is called "Bayes Error". 👇

1⃣ Bayes Optimal Error
- It is the lowest possible error rate for any function (Doesn't have to be 0%).
- Bayes error is a theoretical value that can never be surpassed.
- Since it is a theoretical value, it can't be measured but rather estimated using human-level performance.

Humans tend to be quite good at a lot of tasks such as image and speech recognition. In these cases, human-level performance isn't that far from Bayes error and can be used as an approximation of it.
(Human error ≈ Bayes error).

Note that Bayes error (and human error) does not have to be 0% for all tasks.
for example, speech recognition in a noisy environment (difficult to hear clearly).
In that case, it is impossible to generate a 100% accurate transcription, and Bayes error can be equal to 10%.

2⃣ Bias (Underfitting)
- Bias is the difference between the model predictions and the actual values.
- A model with a high bias means that it has underfitted the training data (the model has a high training set error relative to Bayes error).

- Bias mainly occurs when the model is too simple to understand the underlying structure of the training data, and so it misses the relevant relations between the input features and the target outputs.
- Bias is measured by comparing the training set error to Bayes error.

- Avoidable bias is defined as the difference between the Bayes error (or human error) and the training set error.

For example, for some tasks such as image recognition, the human error rate (an approximation of Bayes error) is usually around 0%. So if the model has a training set error of 10%, this means you have an avoidable bias of 10% and your model is underfitting the training data.

For other tasks such as speech recognition in a noisy environment, the human error rate can be around 15% for example. If your model has an 18% training set error, then you only have a 3% avoidable bias, and the model performance is acceptable.

✍️How to reduce Bias (Underfitting)?
1. Select a more complex and powerful model architecture.
2. Feature engineering to create more and better input features.
3. Longer training.
4. Decrease regularization (model constraints).
5. Better optimization algorithm.

3⃣ Variance (Overfitting)
- Variance is the difference between the training set error and the development set error.
- So variance describes the amount that the model predictions will change when different data is used.

- If a model has a high variance, this means that it has overfitted the training data, and can't generalize well to data instances it hasn't seen before.
- High variance (overfitting) happens when the model is too complex relative to the amount and nosiness of the training data.

- A model with a high variance pays too much attention to the training data and does not generalize well to data it has not seen before. As a result, the model can perform really well on the training data but has a high error rate on the development set and test set.

✍️How to reduce variance (overfitting)?
1. Simplify the model architecture.
2. Select a model with fewer parameters and input features.
3. Get more training data (Data augmentation).
4. Clean the data to reduce the noise.
5. regularization (will be discussed in another thread).

📜Practical Examples
🔴cases no. 1:
Bayes error = 1%
training error = 5%
development error = 6%
This model has a high bias (4%) and low variance (1%)

🔴cases no. 2:
Bayes error = 1%
training error = 2%
development error = 7%
This model has a low bias (1%) and high variance (5%)

🔴cases no. 3:
human error = 1%
training error = 5%
development error = 10%
This model has a high bias (4%) and high variance (5%)

🔴cases no. 4:
human error = 1%
training error = 2%
development error = 3.5%
This model has a low bias (1%) and low variance (1.5%)

Final Notes
a. A model's performance usually slows down once it approaches human-level performance.
b. Before building a machine learning model to solve a particular problem, you should ask about the current solution to give you a good reference for your model performance.

That's it for this thread.

For more content about #MachineLearning and #DataScience, follow me @ammaryh92.