Thread by @TivadarDanka

In machine learning, the inner product (or dot product) of vectors is often used to measure similarity.

However, the formula is far from revealing. What does the sum of coordinate products have to do with similarity?

There is a very simple geometric explanation!

🧵 👇🏽

There are two key things to observe.

First, the inner product is linear in both variables. This property is called bilinearity.

Second, is that the inner product is zero if the vectors are orthogonal.

With these, given an 𝑦, we can decompose 𝑥 into two components: one is orthogonal, while the other is parallel to 𝑦.

So, because of the bilinearity, the inner product equals to the inner product of 𝑦 and the parallel component of 𝑥.

If we write 𝑦 as a scalar multiple of 𝑥, we can see that their inner product can be expressed in terms of the magnitude of 𝑦 and the scalar.

In addition, if we assume that 𝑥 and 𝑦 have unit magnitude, the inner product is even simpler: it describes the scaling factor between 𝑦 and the orthogonal projection of 𝑥 onto 𝑦.

Note that this factor is in [-1, 1]. (It is negative if the directions are opposite.)

There is more. Now comes the really interesting part!

Let's denote the angle between 𝑥 and 𝑦 by α. The scaling factor r equals the cosine of α!

(Recall that we assume that 𝑥 and 𝑦 have unit magnitude.)

If the vectors don't have unit magnitude, we can simply scale them.

The inner product of the scaled vectors is called cosine similarity.

This is probably how you know this quantity. Now you see why!

If you have liked this thread, consider following me and hitting a like/retweet on the first tweet of the thread!

I regularly post simple explanations of seemingly complicated concepts in machine learning, make sure you don't miss out on the next one!