A team led by MIT examined 10 of the most-cited datasets used to test machine learning systems.
They found that around 3.4% of the data was inaccurate or mislabeled.
Those are very popular datasets. How about yours?
↓
They found that around 3.4% of the data was inaccurate or mislabeled.
Those are very popular datasets. How about yours?
↓
I've worked with many datasets for image classification.
Unfortunately, mislabeled data is a common problem.
It is hard for people to consistently label visual concepts, especially when the answer is not apparent.
Unfortunately, mislabeled data is a common problem.
It is hard for people to consistently label visual concepts, especially when the answer is not apparent.
This is a big problem.
Basically, we are evaluating models with images of elephants, expecting them to get classified as "lions."
Your model can't perform well this way.
Basically, we are evaluating models with images of elephants, expecting them to get classified as "lions."
Your model can't perform well this way.
A nice trick to improve your labels:
1. Train a model
2. Evaluate it
3. Review the model mistakes
4. Repeat
Let's dive into this process a little bit more.
1. Train a model
2. Evaluate it
3. Review the model mistakes
4. Repeat
Let's dive into this process a little bit more.
When I finish training a model, I use the validation set and print out the following for each sample:
• Target (what I expect to get)
• Prediction (what the model returned)
• Confidence (the softmax value)
• Target (what I expect to get)
• Prediction (what the model returned)
• Confidence (the softmax value)
I then filter the list by the model mistakes (target and prediction are different) sorted by confidence.
This gives you the mistakes where the model was really confident. ← These usually reveal mislabeled data.
This gives you the mistakes where the model was really confident. ← These usually reveal mislabeled data.
You can also filter the labels with the most significant number of mistakes and start your research from that point.
You want to look into labels with many mistakes because the model is clearly not learning them correctly.
You want to look into labels with many mistakes because the model is clearly not learning them correctly.
After each round of sanitizing the dataset, you can repeat the entire process:
1. Train another model
2. Evaluate it
3. Review the mistakes
You want to stop as soon as there aren't any obvious mislabeled examples on the most confident results of the models.
1. Train another model
2. Evaluate it
3. Review the mistakes
You want to stop as soon as there aren't any obvious mislabeled examples on the most confident results of the models.
Although this process can help improve the quality of your dataset, there's something important you can't forget:
Standardizing the labeling process to avoid mistakes in the first place is one of the best investments you can make.
Standardizing the labeling process to avoid mistakes in the first place is one of the best investments you can make.
Every week, I post 2 or 3 threads like this, breaking down machine learning concepts and giving you ideas on applying them in real-life situations.
You can find more of these at @svpino.
If you find this helpful, stay tuned: a lot more is coming.
You can find more of these at @svpino.
If you find this helpful, stay tuned: a lot more is coming.
If the model is looking at a picture that is supposed to represent a dog (target value = "dog") and it returns with high confidence that it is a "cat," you should look into it:
1. Maybe the model is wrong.
2. Most likely, the picture was mislabeled.
1. Maybe the model is wrong.
2. Most likely, the picture was mislabeled.
https://twitter.com/nelson94008731/status/1443939322516758529?s=20
You will do this process with your validation data.
https://twitter.com/HaSN_CH/status/1443960918023544835?s=20
• • •
Missing some Tweet in this thread? You can try to
force a refresh
