Using more features from your data never comes for free.
Let's talk about dimensionality.
↓
Let's talk about dimensionality.
↓
2. Two days ago I asked this question.
Let's now analyze each option starting with Option 3 (probably the easiest one we can discard.)
Let's now analyze each option starting with Option 3 (probably the easiest one we can discard.)
3. Option 3 states that when we cut down the number of features, we need to "make up the difference" by adding more data.
Removing features reduces the number of dimensions in our data.
It concentrates the samples we have in a lower-dimensional space.
Removing features reduces the number of dimensions in our data.
It concentrates the samples we have in a lower-dimensional space.
4. We can't replace the information provided by a feature with more data.
Cutting down a feature might make it harder for an algorithm to learn our data, but adding more samples won't necessarily solve that.
Option 3 is not a valid answer.
Cutting down a feature might make it harder for an algorithm to learn our data, but adding more samples won't necessarily solve that.
Option 3 is not a valid answer.
5. There are three choices left, and we can find the correct answer using the same insight.
Let's do a mental experiment: Imagine graphing a set of numbers.
Since you have only one dimension, they will all lie somewhere in a line.
Let's do a mental experiment: Imagine graphing a set of numbers.
Since you have only one dimension, they will all lie somewhere in a line.
6. Don't add any new values, but increase the features by adding a second dimension.
Now your values became a set of 2D coordinates (x, y).
If you graph them, they will all be somewhere in a plane.
Now your values became a set of 2D coordinates (x, y).
If you graph them, they will all be somewhere in a plane.
7. If you compare the 1D line with the 2D plane (or even a 3D space assuming you add a third dimension,) something will become apparent quick:
As we increase the dimensionality of the data, it will be harder and harder to fill up the space with the same points.
As we increase the dimensionality of the data, it will be harder and harder to fill up the space with the same points.
8. This increase in sparsity will make it much harder for the learning algorithm to find any interesting patterns.
How can we separate the data with too many dimensions but few samples?
How can we separate the data with too many dimensions but few samples?
9. Based on this, we are ready to make two statements:
1. There's a relationship between features and samples.
2. The more features we add, the more samples we need.
1. There's a relationship between features and samples.
2. The more features we add, the more samples we need.
10. Option 4 is not correct because it violates our first statement above. Option 2 states the opposite of the second statement, so it is also not correct.
Option 1 is the correct solution to this question.
Option 1 is the correct solution to this question.
11. For a more formal definition, look at the Curse of Dimensionality:
The amount of data needed to extract any relevant information increases exponentially with the number of features in your dataset.
The amount of data needed to extract any relevant information increases exponentially with the number of features in your dataset.
Are you into machine learning?
I write practical tips, break down complex concepts, and regularly publish short quizzes to keep you on your toes.
Follow me @svpino and let's do this together!
I write practical tips, break down complex concepts, and regularly publish short quizzes to keep you on your toes.
Follow me @svpino and let's do this together!
• • •
Missing some Tweet in this thread? You can try to
force a refresh
