Machine Learning in the Real World ๐ง ๐ค
ML for real-world applications is much more than designing fancy networks and fine-tuning parameters.
In fact, you will spend most of your time curating a good dataset.
Let's go through the process together ๐
#RepostFriday
ML for real-world applications is much more than designing fancy networks and fine-tuning parameters.
In fact, you will spend most of your time curating a good dataset.
Let's go through the process together ๐
#RepostFriday
Collect Data ๐ฝ
We need to represent the real world as accurately as possible. If some situations are underrepresented we are introducing Sampling Bias.
Sampling Bias is nasty because we'll have high test accuracy, but our model will perform badly when deployed.
๐
We need to represent the real world as accurately as possible. If some situations are underrepresented we are introducing Sampling Bias.
Sampling Bias is nasty because we'll have high test accuracy, but our model will perform badly when deployed.
๐
Traffic Lights ๐ฆ
Let's build a model to recognize traffic lights for a self-driving car. We need to collect data for different:
โช๏ธ Lighting conditions
โช๏ธ Weather conditions
โช๏ธ Distances and viewpoints
โช๏ธ Strange variants
And if we sample only ๐ฆ we won't detect ๐ฅ ๐คทโโ๏ธ
๐
Let's build a model to recognize traffic lights for a self-driving car. We need to collect data for different:
โช๏ธ Lighting conditions
โช๏ธ Weather conditions
โช๏ธ Distances and viewpoints
โช๏ธ Strange variants
And if we sample only ๐ฆ we won't detect ๐ฅ ๐คทโโ๏ธ
๐
Data Cleaning ๐งน
Now we need to clean all corrupted and irrelevant samples. We need to remove:
โช๏ธ Overexposed or underexposed images
โช๏ธ Images in irrelevant situations
โช๏ธ Faulty images
Leaving them in the dataset will hurt our model's performance!
๐
Now we need to clean all corrupted and irrelevant samples. We need to remove:
โช๏ธ Overexposed or underexposed images
โช๏ธ Images in irrelevant situations
โช๏ธ Faulty images
Leaving them in the dataset will hurt our model's performance!
๐
Preprocess Data โ๏ธ
Most ML models like their data nicely normalized and properly scaled. Bad normalization can also lead to worse performance (I have a nice story for another time...)
โช๏ธ Crop and resize all images
โช๏ธ Normalize all values (usually 0 mean and 1 std. dev.)
๐
Most ML models like their data nicely normalized and properly scaled. Bad normalization can also lead to worse performance (I have a nice story for another time...)
โช๏ธ Crop and resize all images
โช๏ธ Normalize all values (usually 0 mean and 1 std. dev.)
๐
Label Data ๐ท๏ธ
Manual labeling is expensive. Try to be clever and automate as much as possible:
โช๏ธ Generate labels from the input data
โช๏ธ Use slow, but accurate algorithms offline
โช๏ธ Pre-label data during collection
โช๏ธ Develop good labeling tools
โช๏ธ Use synthetic data?
๐
Manual labeling is expensive. Try to be clever and automate as much as possible:
โช๏ธ Generate labels from the input data
โช๏ธ Use slow, but accurate algorithms offline
โช๏ธ Pre-label data during collection
โช๏ธ Develop good labeling tools
โช๏ธ Use synthetic data?
๐
Label Correction โ
You will always have errors in the labels - humans make mistakes. Review and iterate!
โช๏ธ Spot checks to find systematic problems
โช๏ธ Improve labeling guidelines and tools
โช๏ธ Review test results and fix labels
โช๏ธ Label samples multiple times
๐
You will always have errors in the labels - humans make mistakes. Review and iterate!
โช๏ธ Spot checks to find systematic problems
โช๏ธ Improve labeling guidelines and tools
โช๏ธ Review test results and fix labels
โช๏ธ Label samples multiple times
๐
The danger of label errors ๐งโ๐ซ
A recent study by MIT found that 10 of the most popular public datasets had 3.4% label errors on average (ImageNet had 5.8%).
This even lead authors to choose the wrong (and more complex) model as their best one!
arxiv.org/abs/2103.14749
๐
A recent study by MIT found that 10 of the most popular public datasets had 3.4% label errors on average (ImageNet had 5.8%).
This even lead authors to choose the wrong (and more complex) model as their best one!
arxiv.org/abs/2103.14749
๐
Balance Dataset โ๏ธ
Dealing with imbalanced data can be tricky...
Let's classify the color of the ๐ฆ - we can get 97% just by learning to recognize ๐ข and ๐ด, just because ๐ก is severely underrepresented.
I have a separate thread on this topic:
๐
Dealing with imbalanced data can be tricky...
Let's classify the color of the ๐ฆ - we can get 97% just by learning to recognize ๐ข and ๐ด, just because ๐ก is severely underrepresented.
I have a separate thread on this topic:
https://twitter.com/haltakov/status/1359643905973760001
๐
Train and Evaluate Model ๐ช๐
This is the part that is usually covered by ML courses. Now is the time to try out different features, network architectures, fine-tune parameters etc.
But we are not done yet... ๐
This is the part that is usually covered by ML courses. Now is the time to try out different features, network architectures, fine-tune parameters etc.
But we are not done yet... ๐
Iterative Process ๐
In most real-world applications the bottleneck is not the model itself, but the data. After having a first model, we need to review where it has problems and go back to:
โช๏ธ Collecting and labeling more data
โช๏ธ Correcting labels
โช๏ธ Balancing the data
๐
In most real-world applications the bottleneck is not the model itself, but the data. After having a first model, we need to review where it has problems and go back to:
โช๏ธ Collecting and labeling more data
โช๏ธ Correcting labels
โช๏ธ Balancing the data
๐
Deploy Model ๐ข
Deploying the model in production poses some additional constraints:
โช๏ธ Speed
โช๏ธ Cost
โช๏ธ Stability
โช๏ธ Privacy
โช๏ธ Hardware availability and integration
We have to find a good trade-off between these factors and accuracy.
Now we are done, right? No...๐
Deploying the model in production poses some additional constraints:
โช๏ธ Speed
โช๏ธ Cost
โช๏ธ Stability
โช๏ธ Privacy
โช๏ธ Hardware availability and integration
We have to find a good trade-off between these factors and accuracy.
Now we are done, right? No...๐
Monitoring ๐ฅ๏ธ
The performance of the model will start degrading over time because the world keeps changing:
โช๏ธ Concept drift - the real-world distribution changes
โช๏ธ Data drift - the properties of the data change
We need to detect this, retrain, and deploy again.
Example ๐
The performance of the model will start degrading over time because the world keeps changing:
โช๏ธ Concept drift - the real-world distribution changes
โช๏ธ Data drift - the properties of the data change
We need to detect this, retrain, and deploy again.
Example ๐
Drift โก๏ธ
We now have a trained model to recognize ๐ฆ, but people keep inventing new variants - see what some creative people in Munich came up with ๐
We need a way to detect that we have a problem, collect data, label, and retrain our model.
๐
We now have a trained model to recognize ๐ฆ, but people keep inventing new variants - see what some creative people in Munich came up with ๐
We need a way to detect that we have a problem, collect data, label, and retrain our model.
๐
Summary ๐
This is how a typical ML pipeline for real-world applications looks like. Please remember this:
โช๏ธ Curating a good dataset is the most important thing
โช๏ธ Dataset curation is an iterative process
โช๏ธ Monitoring is critical to ensure good performance over time
This is how a typical ML pipeline for real-world applications looks like. Please remember this:
โช๏ธ Curating a good dataset is the most important thing
โช๏ธ Dataset curation is an iterative process
โช๏ธ Monitoring is critical to ensure good performance over time
Every Friday I repost one of my old threads so more people get the chance to see them. During the rest of the week, I post new content on machine learning and web3.
If you are interested in seeing more, follow me @haltakov
.
If you are interested in seeing more, follow me @haltakov
.
โข โข โข
Missing some Tweet in this thread? You can try to
force a refresh
ใ
