A neural network doesn't know when it doesn't know.
If you think about it, recognizing when a data point is absolutely unlike any other previously seen is a problem rarely dealt with.
However, it is essential.
In this thread, I'll explain how and why!
🧵 👇🏽
If you think about it, recognizing when a data point is absolutely unlike any other previously seen is a problem rarely dealt with.
However, it is essential.
In this thread, I'll explain how and why!
🧵 👇🏽
Suppose that this is your training data.
The situation looks fairly straightforward: a simple logistic regression solves the problem.
The model is deployed to production without a second thought.
Now comes the surprise!
The situation looks fairly straightforward: a simple logistic regression solves the problem.
The model is deployed to production without a second thought.
Now comes the surprise!
We start receiving new data for prediction when we see the following pattern emerge.
The new instances are confidently classified, incorrectly.
The new instances are confidently classified, incorrectly.
Situations like these happen all the time.
For instance, you are building an image-based tool for retailers to detect products on a shelf. One day, some manufacturer comes out with a new product, in new packaging.
How do you prepare your system for that?
For instance, you are building an image-based tool for retailers to detect products on a shelf. One day, some manufacturer comes out with a new product, in new packaging.
How do you prepare your system for that?
One answer is open set recognition: detecting the unknowns.
Instead of looking at the class probabilities given by our network, we take a single step back and analyze the distribution of the resulting feature vector.
For a single feature, it may look something like this.
Instead of looking at the class probabilities given by our network, we take a single step back and analyze the distribution of the resulting feature vector.
For a single feature, it may look something like this.
The method called OpenMax (which is just one among the many, but was one of the first algorithms) detects these anomalies by fitting a particular distribution to the features and checks if new values fit.
In the paper Towards Open Set Deep Networks by Abhijit Bendale and Terrance E. Boult, the authors demonstrate how revealing the activation distributions are.
In their illustration below, you can see that new examples do indeed have different distributions.
In their illustration below, you can see that new examples do indeed have different distributions.
This method works if you don't have any control over the training.
Can we perform better if we can modify the training? What if we train the network to be prepared for open set recognition tasks from the start?
Turns out, we can do this.
Can we perform better if we can modify the training? What if we train the network to be prepared for open set recognition tasks from the start?
Turns out, we can do this.
In their paper Reducing Network Agnostophobia, the authors Akshay Raj Dhamija, Manuel Günther, and Terrance E. Boult came up with a method that can do this.
They observed that the activation of unknown examples tends to cluster around the origin.
arxiv.org/pdf/1811.04110…
They observed that the activation of unknown examples tends to cluster around the origin.
arxiv.org/pdf/1811.04110…
Their key idea was that with adding an "unknown" class and using a special loss function, the norm of unknown samples is forced to zero.
This is called Objectosphere loss. Below is the author's illustration of its performance on MNIST digit recognition.
This is called Objectosphere loss. Below is the author's illustration of its performance on MNIST digit recognition.
Although open set recognition is an old problem, it hasn't seen much research activity until recently.
Since machine learning algorithms are not constrained to laboratory settings anymore, it is essential to perform this task properly.
Since machine learning algorithms are not constrained to laboratory settings anymore, it is essential to perform this task properly.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
