Softmax is one of the most commonly used functions in machine learning.
It is used to transform high-level features into probabilities. Based on the formula, it is hard to imagine how it is done exactly.
Softmax might not be what you think it is. Let's find out why!
🧵 👇🏽
It is used to transform high-level features into probabilities. Based on the formula, it is hard to imagine how it is done exactly.
Softmax might not be what you think it is. Let's find out why!
🧵 👇🏽
First, we start with the exponential function eˣ, which transforms a real number into a positive one.
It has a feature that shows the geometry of this transformation: it turns addition into multiplication.
In particular, eᵃ ⁺ ᵇ = eᵃ eᵇ holds.
It has a feature that shows the geometry of this transformation: it turns addition into multiplication.
In particular, eᵃ ⁺ ᵇ = eᵃ eᵇ holds.
The input x = (x₁, x₂, ..., xₙ) consists of the highest level features: the class scores.
For two vectors x and y, xᵢ - yᵢ expresses the difference between features.
After the exponential function, this is transformed into their ratio.
For two vectors x and y, xᵢ - yᵢ expresses the difference between features.
After the exponential function, this is transformed into their ratio.
To turn these transformed features into probabilities, we have to normalize them so that the sum is one.
This is why we divide with the sum of all exponentials in the definition of Softmax.
This is why we divide with the sum of all exponentials in the definition of Softmax.
Although Softmax is very popular, there is a big issue with it.
Since exponentiation turns addition into multiplication, normalizing causes the function to map very different feature vectors to the same probability distributions.
Since exponentiation turns addition into multiplication, normalizing causes the function to map very different feature vectors to the same probability distributions.
If the features represent the class labels "cat", "dog", "kangaroo", the vector (1, 2, 3) intuitively means that it is a bit more likely "kangaroo" than the others.
(-10, -9, -8) signals that the input is neither. Yet, their Softmax is the same.
Why?
(-10, -9, -8) signals that the input is neither. Yet, their Softmax is the same.
Why?
Because Softmax is invariant for translating each feature with the same value.
If this is not clear, check out the calculation below.
If this is not clear, check out the calculation below.
This is the reason why prediction uncertainty is hard to estimate with Softmax.
Overall, you have to be careful when using Softmax. Instead of thinking about the output as the true class probability distribution, view it as an estimation for which class is the most likely.
Overall, you have to be careful when using Softmax. Instead of thinking about the output as the true class probability distribution, view it as an estimation for which class is the most likely.
If you enjoyed this thread, consider following me and hitting a like/retweet on the first tweet of the thread!
I regularly post simple explanations of seemingly complicated concepts in machine learning, make sure you don't miss out on the next one!
I regularly post simple explanations of seemingly complicated concepts in machine learning, make sure you don't miss out on the next one!
An important point by @tpk_in: the exponential function can lead to overflow, so the translational invariance mentioned above can be useful for implementations.
https://twitter.com/tpk_in/status/1384386555779829764
• • •
Missing some Tweet in this thread? You can try to
force a refresh
