Here is the story of one of those hidden issues with machine learning models that books don't tell you about.
This happened in real life: ↓
This happened in real life: ↓
Imagine you are building a computer vision model.
It goes something like this:
1. Load a dataset of images
2. Train a model with those images
3. Export the final model
Pretty standard stuff.
It goes something like this:
1. Load a dataset of images
2. Train a model with those images
3. Export the final model
Pretty standard stuff.
To make it more specific, let's imagine that you are using OpenCV to load the images from the disk.
Something like the attached screenshot.
Nothing fancy here, right?
Something like the attached screenshot.
Nothing fancy here, right?
Your model trains. Your model is excellent. You are ready to deploy your model.
You take it and host it somewhere.
Life is good!
Except...
You take it and host it somewhere.
Life is good!
Except...
Your model starts making predictions... and they are terrible.
You weren't expecting this at all.
What's going on? You validated and tested your model, and it did great. Why would predictions be all over the place?
It took a few days to figure this out.
You weren't expecting this at all.
What's going on? You validated and tested your model, and it did great. Why would predictions be all over the place?
It took a few days to figure this out.
If you have done this before, I'm sure you have a checklist you'd suggest going through.
I checked off every item from mine, and still nothing!
I wasn't expecting the problem that we found.
I checked off every item from mine, and still nothing!
I wasn't expecting the problem that we found.
Here is something that I didn't tell you before:
The operating system we used to train our model was different from the system where we deployed it.
But, why would this be a problem?
The operating system we used to train our model was different from the system where we deployed it.
But, why would this be a problem?
Turns out that Open CV relies on the underlying operating system codecs to load compressed images like JPEG files.
So, whenever you call "cv2.imread()", you ask the operating system to load that image for you.
Different codecs return slightly different pixel values.
So, whenever you call "cv2.imread()", you ask the operating system to load that image for you.
Different codecs return slightly different pixel values.
We were training a model with a set of values and using a different set to make predictions.
The differences were enough to screw up those predictions.
All of this because the operating systems were different!
The differences were enough to screw up those predictions.
All of this because the operating systems were different!
Yes, of course, Windows was involved.
Linux-based systems seem to handle images the same way, at least in our tests.
The solution: Make sure your training and prediction environments are the same. ← This one now goes into my checklist.
Linux-based systems seem to handle images the same way, at least in our tests.
The solution: Make sure your training and prediction environments are the same. ← This one now goes into my checklist.
Yes, containers solve this problem.
https://twitter.com/blockchain092/status/1455491089205432321
• • •
Missing some Tweet in this thread? You can try to
force a refresh
