Machine Learning in the Real World π§ π€
ML for real-world applications is much more than designing fancy networks and fine-tuning parameters.
In fact, you will spend most of your time curating a good dataset.
Let's go through the steps of the process together π
ML for real-world applications is much more than designing fancy networks and fine-tuning parameters.
In fact, you will spend most of your time curating a good dataset.
Let's go through the steps of the process together π
Collect Data π½
We need to represent the real world as accurately as possible. If some situations are underrepresented we are introducing Sampling Bias.
Sampling Bias is nasty because we'll have high test accuracy, but our model will perform badly when deployed.
π
We need to represent the real world as accurately as possible. If some situations are underrepresented we are introducing Sampling Bias.
Sampling Bias is nasty because we'll have high test accuracy, but our model will perform badly when deployed.
π
Traffic Lights π¦
Let's build a model to recognize traffic lights for a self-driving car. We need to collect data for different:
βͺοΈ Lighting conditions
βͺοΈ Weather conditions
βͺοΈ Distances and viewpoints
βͺοΈ Strange variants
And if we sample only π¦ we won't detect π₯ π€·ββοΈ
π
Let's build a model to recognize traffic lights for a self-driving car. We need to collect data for different:
βͺοΈ Lighting conditions
βͺοΈ Weather conditions
βͺοΈ Distances and viewpoints
βͺοΈ Strange variants
And if we sample only π¦ we won't detect π₯ π€·ββοΈ
π
Data Cleaning π§Ή
Now we need to clean all corrupted and irrelevant samples. We need to remove:
βͺοΈ Overexposed or underexposed images
βͺοΈ Images in irrelevant situations
βͺοΈ Faulty images
Leaving them in the dataset will hurt our model's performance!
π
Now we need to clean all corrupted and irrelevant samples. We need to remove:
βͺοΈ Overexposed or underexposed images
βͺοΈ Images in irrelevant situations
βͺοΈ Faulty images
Leaving them in the dataset will hurt our model's performance!
π
Preprocess Data βοΈ
Most ML models like their data nicely normalized and properly scaled. Bad normalization can also lead to worse performance (I have a nice story for another time...)
βͺοΈ Crop and resize all images
βͺοΈ Normalize all values (usually 0 mean and 1 std. dev.)
π
Most ML models like their data nicely normalized and properly scaled. Bad normalization can also lead to worse performance (I have a nice story for another time...)
βͺοΈ Crop and resize all images
βͺοΈ Normalize all values (usually 0 mean and 1 std. dev.)
π
Label Data π·οΈ
Manual labeling is expensive. Try to be clever and automate as much as possible:
βͺοΈ Generate labels from the input data
βͺοΈ Use slow, but accurate algorithms offline
βͺοΈ Pre-label data during collection
βͺοΈ Develop good labeling tools
βͺοΈ Use synthetic data?
π
Manual labeling is expensive. Try to be clever and automate as much as possible:
βͺοΈ Generate labels from the input data
βͺοΈ Use slow, but accurate algorithms offline
βͺοΈ Pre-label data during collection
βͺοΈ Develop good labeling tools
βͺοΈ Use synthetic data?
π
Label Correction β
You will always have errors in the labels - humans make mistakes. Review and iterate!
βͺοΈ Spot checks to find systematic problems
βͺοΈ Improve labeling guidelines and tools
βͺοΈ Review test results and fix labels
βͺοΈ Label samples multiple times
π
You will always have errors in the labels - humans make mistakes. Review and iterate!
βͺοΈ Spot checks to find systematic problems
βͺοΈ Improve labeling guidelines and tools
βͺοΈ Review test results and fix labels
βͺοΈ Label samples multiple times
π
The danger of label errors π§βπ«
A recent study by MIT found that 10 of the most popular public datasets had 3.4% label errors on average (ImageNet had 5.8%).
This even lead authors to choose the wrong (and more complex) model as their best one!
arxiv.org/abs/2103.14749
π
A recent study by MIT found that 10 of the most popular public datasets had 3.4% label errors on average (ImageNet had 5.8%).
This even lead authors to choose the wrong (and more complex) model as their best one!
arxiv.org/abs/2103.14749
π
Balance Dataset βοΈ
Dealing with imbalanced data can be tricky...
Let's classify the color of the π¦ - we can get 97% just by learning to recognize π’ and π΄, just because π‘ is severely underrepresented.
I have a separate thread on this topic:
π
Dealing with imbalanced data can be tricky...
Let's classify the color of the π¦ - we can get 97% just by learning to recognize π’ and π΄, just because π‘ is severely underrepresented.
I have a separate thread on this topic:
https://twitter.com/haltakov/status/1359643905973760001
π
Train and Evaluate Model πͺπ
This is the part that is usually covered by ML courses. Now is the time to try out different features, network architectures, fine-tune parameters etc.
But we are not done yet... π
This is the part that is usually covered by ML courses. Now is the time to try out different features, network architectures, fine-tune parameters etc.
But we are not done yet... π
Iterative Process π
In most real-world applications the bottleneck is not the model itself, but the data. After having a first model, we need to review where it has problems and go back to:
βͺοΈ Collecting and labeling more data
βͺοΈ Correcting labels
βͺοΈ Balancing the data
π
In most real-world applications the bottleneck is not the model itself, but the data. After having a first model, we need to review where it has problems and go back to:
βͺοΈ Collecting and labeling more data
βͺοΈ Correcting labels
βͺοΈ Balancing the data
π
Deploy Model π’
Deploying the model in production poses some additional constraints:
βͺοΈ Speed
βͺοΈ Cost
βͺοΈ Stability
βͺοΈ Privacy
βͺοΈ Hardware availability and integration
We have to find a good trade-off between these factors and accuracy.
Now we are done, right? No...π
Deploying the model in production poses some additional constraints:
βͺοΈ Speed
βͺοΈ Cost
βͺοΈ Stability
βͺοΈ Privacy
βͺοΈ Hardware availability and integration
We have to find a good trade-off between these factors and accuracy.
Now we are done, right? No...π
Monitoring π₯οΈ
The performance of the model will start degrading over time because the world keeps changing:
βͺοΈ Concept drift - the real-world distribution changes
βͺοΈ Data drift - the properties of the data change
We need to detect this, retrain, and deploy again.
Example π
The performance of the model will start degrading over time because the world keeps changing:
βͺοΈ Concept drift - the real-world distribution changes
βͺοΈ Data drift - the properties of the data change
We need to detect this, retrain, and deploy again.
Example π
Drift β‘οΈ
We now have a trained model to recognize π¦, but people keep inventing new variants - see what some creative people in Munich came up with π
We need a way to detect that we have a problem, collect data, label, and retrain our model.
π
We now have a trained model to recognize π¦, but people keep inventing new variants - see what some creative people in Munich came up with π
We need a way to detect that we have a problem, collect data, label, and retrain our model.
π
Summary π
This is how a typical ML pipeline for real-world applications looks like. Please remember this:
βͺοΈ Curating a good dataset is the most important thing
βͺοΈ Dataset curation is an iterative process
βͺοΈ Monitoring is critical to ensure good performance over time
This is how a typical ML pipeline for real-world applications looks like. Please remember this:
βͺοΈ Curating a good dataset is the most important thing
βͺοΈ Dataset curation is an iterative process
βͺοΈ Monitoring is critical to ensure good performance over time
This week I'm reposting some of my best threads from the past months, so I can focus on creating my machine learning course.
Next week I'm back with some new content on machine learning and web3, so make sure you follow me @haltakov.
Next week I'm back with some new content on machine learning and web3, so make sure you follow me @haltakov.
β’ β’ β’
Missing some Tweet in this thread? You can try to
force a refresh
γ
