Real world datasets are often imbalanced - some of the classes appear much more often in your data than others.
The problem? You ML model will likely learn to only predict the dominant classes.
What can you do about it? π€
Thread π
Example π¦
We will be dealing with a ML model to detect traffic lights for a self-driving car π€π
Traffic lights are small so you will have much more parts of the image that are not traffic lights.
Furthermore, yellow lights π‘ are much rarer than green π’ or red π΄.
The problem β‘
Imagine we train a model to classify the color of the traffic light. A typical distribution will be:
π΄ - 56%
π‘ - 3%
π’ - 41%
So, your model can get to 97% accuracy just by learning to distinguish red from green.
How can we deal with this? π€
Evaluation measures π
First, you need to start using a different evaluation measure than accuracy:
- Precision per class
- Recall per class
- F1 score per class
I also like to look at the confusion matrix to get an overview. Always look at examples from the data as well!
In the traffic lights example above, we will see very poor recall for π‘ (most real examples were not recognized), while precision will likely be high.
At the same time, the precision of π’ and π΄ will be lower (π‘ will be classified as π’ or π΄).
Get more data π’
The best thing you can do is to collect more data of the underrepresented classes. This may be hard or even impossible...
You can imagine ways to record more yellow lights, but imagine you want to detect a very rare disease in CT images?
Balance your data π
The idea is to resample your dataset so it is better balanced.
βͺοΈUndersampling - throw away some examples of the dominant classes
βͺοΈ Oversampling - get more samples of the underrepresented class
Undersampling β¬
The easiest way is to just randomly throw away samples from the dominant class.
Even better, you can use some unsupervised clustering method and throw out only samples from the big clusters.
The problem of course is that you are throwing out valuable data...
Oversampling β«
This is more difficult. You can just repeat sample, but it won't work very good.
You can use methods like SMOTE (Synthetic Minority Oversampling Technique) to generate new samples interpolating between existing ones. This may not be easy for complex images.
Oversampling β«
If you are dealing with images, you can use data augmentation techniques to create new samples by modifying the existing ones (rotation, flipping, skewing, color filters...)
You can also use GANs or simulation the synthesize completely new images.
Adapting your loss π
Another strategy is to modify your loss function to penalize misclassification of the underrepresented classes more than the dominant ones.
In the π¦ example we can set them like this (proportionally to the distribution)
π΄ - 1.8
π‘ - 33.3
π’ - 2.4
If you are training a neural network with TensorFlow or PyTorch you can do this very easily:
This is the formula for Mean Squared Error (MSE) as defined in WikiPedia. It represents a very simple concept, but may not be easy to read if you are just starting with ML.
Read below and it will be a piece of cake! π°
Thread π
The core β«
Let's unpack from the inside out. MSE calculates how close are your model's predictions ΕΆ to the ground truth labels Y. You want the error to go to 0.
If you are predicting house prices, the error could be the difference between the predicted and the actual price.
Why squared? 2οΈβ£
Subtracting the prediction from the label won't work. The error may be negative or positive, which is a problem when summing up samples.
You can take the absolute value or the square of the error. The square has the property that it punished bigger errors more.
What are typical challenges when training a deep neural networks βοΈ
βͺοΈ Overfitting
βͺοΈ Underfitting
βͺοΈ Lack of training data
βͺοΈ Vanishing gradients
βͺοΈ Exploding gradients
βͺοΈ Dead ReLUs
βͺοΈ Network architecture design
βͺοΈ Hyperparameter tuning
How to solve them π
Overfitting π
Your model performs well during training, but poorly during test.
Possible solutions:
- Reduce the size of your model
- Add more data
- Increase dropout
- Stop the training early
- Add regularization to your loss
- Decrease batch size
Underfitting π
You model performs poorly both during training and test.
Possible solutions:
- Increase the size of your model
- Add more data
- Train for a longer time
- Start with a pre-trained network