When we start with machine learning, we learn to split our datasets in testing and training by taking a percentage of the data.
Unfortunately, this practice could lead to overestimating the performance of your model.
↓ 1/7
Unfortunately, this practice could lead to overestimating the performance of your model.
↓ 1/7
Imagine a dataset of pictures with people doing signals with their hands.
As we were told, we take 70% of the images for training and the remaining 30% for testing. We are careful to maintain the original ratio between classes.
How could this be a problem?
↓ 2/7
As we were told, we take 70% of the images for training and the remaining 30% for testing. We are careful to maintain the original ratio between classes.
How could this be a problem?
↓ 2/7
There are a lot of pictures of Mary in the dataset. She is showing different signals with her hands.
Also Joe. He was a model too that participated in the creation of the dataset.
↓ 3/7
Also Joe. He was a model too that participated in the creation of the dataset.
↓ 3/7
By splitting the data without taking this into account, we will get Mary and Joe in both the training and testing sets.
Unfortunately, this is giving our model an unfair advantage doing testing.
↓ 4/7
Unfortunately, this is giving our model an unfair advantage doing testing.
↓ 4/7
In a real-life situation, our model will not see Mary or Joe. However, we both trained and tested with pictures of them.
Ideally, you want to test with a distribution that closely resembles real-life.
↓ 5/7
Ideally, you want to test with a distribution that closely resembles real-life.
↓ 5/7
The solution for this? Train with Mary. Test with Joe.
You need to understand your data. Random splits are dangerous.
↓ 6/7
You need to understand your data. Random splits are dangerous.
↓ 6/7
Follow me for practical tips about machine learning. Those that schools don't teach, and you have to beat your head against the wall before finding out.
Check my feed at @svpino for more.
7/7
Check my feed at @svpino for more.
7/7
K-Fold Cross-validation will give you a better idea of performance than using a single test set.
One problem with this method is that it becomes prohibitive for large datasets.
One problem with this method is that it becomes prohibitive for large datasets.
https://twitter.com/cesarluismeza/status/1377979580577898505?s=20
Think about it this way:
What information you are using to test your model that's unrealistic you'll see in real life?
In this example, we are using both Mary and Joe in our dataset to both train and test the model, but we really need different people.
What information you are using to test your model that's unrealistic you'll see in real life?
In this example, we are using both Mary and Joe in our dataset to both train and test the model, but we really need different people.
https://twitter.com/scienziato_dati/status/1377978626470203392?s=20
If you are expecting to see loan applications from customers that have no record, then yes.
You want your test set to represent reality as close as possible.
You want your test set to represent reality as close as possible.
https://twitter.com/TolgaKa80194996/status/1377978415249248256?s=20
• • •
Missing some Tweet in this thread? You can try to
force a refresh
