Thread by @ammaryh92

The ABSOLUTE ESSENTIALS of Data Mismatch in Machine Learning

🧵This thread will cover the following concepts:
a. Data Distribution (explanation + examples)
b. Data Mismatch Problem

In a previous thread, we discussed that we should split our data into three parts:
- Training Set
- Development Set
- Test Set
👇👇

https://twitter.com/ammaryh92/status/1419321589234024448

📜Ideally, All 3 sets (train, dev, test) should be perfectly representative of the new instances you want to generalize to.
Otherwise, a good performance on the training data will not correspond to doing well on your application.

⚠️However, getting representative data for training is sometimes very difficult.

👉Example
Assume you want to build an app that recognizes the species of flowers in images taken by the users.

For this app, you will need to train a ML algorithm on, let's say, 200,000 images.

You can download millions of flower images from the internet using web scraping to collect the required images.

⚡️The problem is that the images you download will probably be high-resolution, professionally shot images.

On the other hand, the images taken by the app users will probably be low-quality, blurry images.

In that case, the images used to train and evaluate the model's performance will not be representative of the ones used in production.

💡Solutions
1⃣Always make sure that the development set and the test set are perfectly representative of the data you expect to use in production.
2⃣Always make sure that the development set and the test set come from the same distribution.

This ensures that a good performance on the development/test set will correspond to a good performance on production data.

🔴Practical Example
When building your flower app, you collected 200,000 images from the web (non-representative), and 10,000 images taken by a phone camera (representative).
How should you distribute them?
👇👇

📒Remember that the dev/test sets should be made up exclusively from the representative images (10,000).

A Possible solution
1⃣Total images (210,000 images)
200,000 web images (non-representative)+ 10,000 phone images (representative)

2⃣Training set (205,000 images)
200,000 web images (non-representative) + 5,000 phone images (representative).

3⃣Development set
2500 phone images (representative)

4⃣Test set
2500 phone images (representative)

📜The Problem of Data Mismatch
This problem arises when the training data comes from a different distribution than the dev/test data (like in the above example).
After training the model, if the model performs poorly on the dev set, then there are two possible causes:
👇👇

a. Overfitting (high variance)
b. Data mismatch (training data has a different distribution than dev set data)

🤔How to determine the correct cause?
We hold out some of the training images (web images) in another set called the train-dev set.

We then train the model on the training set (not the train-dev set).
Next, we evaluate the model performance on the train-dev set.
If the model performs poorly on the train-dev set, then we do have an overfitting problem.

However, if the model performs well on the train-dev set, but then performs poorly on the development set, then the poor performance is caused by data mismatch.

We can solve the data mismatch problem using preprocessing and data augmentation techniques to make the training data (from the web) look more like the development data, and then retrain the model.

That's it for this thread.

For additional information, I highly recommend reading this amazing article
yashuseth.blog/2018/03/20/wha…

If you found this thread to be useful, kindly consider supporting me by retweeting the first tweet.

For more Data Science and Machine Learning content like this, follow me @ammaryh92.