Machine Learning Explained π¨βπ«
PCA
Principal Component Analysis is a commonly used method for dimensionality reduction.
It's a good example of how fairly complex math can have an intuitive explanation and be easy to use in practice.
Let's start from the application of PCA π
PCA
Principal Component Analysis is a commonly used method for dimensionality reduction.
It's a good example of how fairly complex math can have an intuitive explanation and be easy to use in practice.
Let's start from the application of PCA π
Dimensionality Reduction
This is one of the common uses of PCA in machine learning.
Imagine you want to predict house prices. You get a large table of many houses and different features for them like size, number of rooms, location, age, etc.
Some features seem correlated π
This is one of the common uses of PCA in machine learning.
Imagine you want to predict house prices. You get a large table of many houses and different features for them like size, number of rooms, location, age, etc.
Some features seem correlated π
Correlated features
For example, the size of the house is correlated with the number of rooms. Bigger houses tend to have more rooms.
Another example could be the age and the year the house was built - they give us pretty much the same information.
We don't want that π
For example, the size of the house is correlated with the number of rooms. Bigger houses tend to have more rooms.
Another example could be the age and the year the house was built - they give us pretty much the same information.
We don't want that π
Curse of Dimensionality
In general, we want to have fewer features, because of the Curse of Dimensionality.
The amount of data required to fit a model increases exponentially with the number of features. Therefore, having many features telling us the same thing is bad.
π
In general, we want to have fewer features, because of the Curse of Dimensionality.
The amount of data required to fit a model increases exponentially with the number of features. Therefore, having many features telling us the same thing is bad.
π
Remove features
Then, let's remove the features we don't need. But how do you select these?
Take a look at the code. Using scikit-learn and PCA you can take a dataset containing many features and transform it into a dataset with fewer features (5 in the example).
π
Then, let's remove the features we don't need. But how do you select these?
Take a look at the code. Using scikit-learn and PCA you can take a dataset containing many features and transform it into a dataset with fewer features (5 in the example).
π
What you can use PCA for is to essentially create new features that will "compress" the data of the full feature set as well as possible.
In this way, redundant features will effectively be removed. This is what we call dimensionality reduction.
Why does it work π
In this way, redundant features will effectively be removed. This is what we call dimensionality reduction.
Why does it work π
To understand better what happens, let's look at a specific example using the awesome visualization by @vicapow.
I'll post images and videos in this thread, but I encourage you to click the link and experiment yourself a bit.
setosa.io/ev/principal-cβ¦
Let's start in 3D π
I'll post images and videos in this thread, but I encourage you to click the link and experiment yourself a bit.
setosa.io/ev/principal-cβ¦
Let's start in 3D π
@vicapow Take a look at this 3D model
Guess what, you are now doing dimensionality reduction!
You see, you are looking at a 3D model on a screen that is 2D. While every point in the model is defined by 3 features (coordinates), you only select 2 features to look at it.
π
Guess what, you are now doing dimensionality reduction!
You see, you are looking at a 3D model on a screen that is 2D. While every point in the model is defined by 3 features (coordinates), you only select 2 features to look at it.
π
@vicapow As the model is rotating, you can observe that some views are better at displaying the data than others.
Look at the screenshots I did below. On the first one, you can clearly see that there are 3 clusters in the data, while on the second you only see 2.
Can PCA help? π
Look at the screenshots I did below. On the first one, you can clearly see that there are 3 clusters in the data, while on the second you only see 2.
Can PCA help? π
@vicapow I wrote above that PCA will find a new reduced set of features that optimally encodes the full dataset.
So, if we apply it to the 3D model, we get the following view of the data, which clearly shows the 3 clusters. The optimal view!
π
So, if we apply it to the 3D model, we get the following view of the data, which clearly shows the 3 clusters. The optimal view!
π
@vicapow Now you can imagine that on the 3 dimensions you have features describing a house.
And the good thing is that PCA doesn't care how many dimensions you have and how many you want to reduce your dataset.
Now a bit about the math π
And the good thing is that PCA doesn't care how many dimensions you have and how many you want to reduce your dataset.
Now a bit about the math π
@vicapow What happens is that PCA will rotate your dataset in a way that in the first dimension (first principal component) you have the highest variability of the data, on the second dimension - the second-highest, and so on.
The reduction happens by removing the last dimensions.
π
The reduction happens by removing the last dimensions.
π
@vicapow Now, go back to the visualization and play with the 2D example.
Move the points in the left around and observe how the PCA plot changes. Put all points on a diagonal like (highly correlated) and you will see that after PCA the second dimension almost disappears.
π
Move the points in the left around and observe how the PCA plot changes. Put all points on a diagonal like (highly correlated) and you will see that after PCA the second dimension almost disappears.
π
@vicapow The math
You saw how you can use PCA in practice and got an intuition of why it works and you haven't seen a single formula!
To perform PCA you need to perform Singular Value Decomposition of the covariance matrix, but I won't go into detail here.
π
You saw how you can use PCA in practice and got an intuition of why it works and you haven't seen a single formula!
To perform PCA you need to perform Singular Value Decomposition of the covariance matrix, but I won't go into detail here.
π
@vicapow If you are interested in math let me know and I can write another thread.
However, I really want to stress that you can understand PCA and use it in practice even without understanding the internals of how it works.
π
However, I really want to stress that you can understand PCA and use it in practice even without understanding the internals of how it works.
π
@vicapow I regularly write threads to explain complex concepts in machine learning and web3 in a simple manner.
Follow me @haltakov for more
Follow me @haltakov for more
β’ β’ β’
Missing some Tweet in this thread? You can try to
force a refresh
γ
