Thread by @svpino

Today, let's talk about two key data transformations we constantly use in machine learning:

▫️ Label encoding
▫️ One-hot-encoding

But let's not just talk about them, but try to build some intuition about why they are important.

Grab a coffee, and let's start! ☕️🧵👇

Imagine we have a dataset with two features:

▫️ "temperature" — a numeric value.
▫️ "weather" — a string value.

You should feel uncomfortable with this dataset right off the bat: machine learning algorithms usually don't like to work with non-numerical data.

[2 / 15]

To set the record straight, some algorithms don't mind non-numerical data.

For example, certain Decision Tree implementations will be fine with the "weather" feature from our example.

But a lot of them can only work with numbers.

[3 / 15]

Let's look closely at the weather feature. You'll notice there are only three possible values: sunny, overcast, and rainy.

We call these "categorical features." Basically, they are features that can only take a limited number of possible values.

[4 / 15]

Alright, so we want to change this feature, and we know there are only three possible values that it can take.

Let's do the obvious thing: replace each string value with a number.

Now the whole dataset is numeric, so algorithms shouldn't complain.

[5 / 15]

The process of converting a categorical feature to a numerical feature is called "label encoding."

Here is a Python 🐍 example that encodes our "weather" feature.

[6 / 15]

Why that specific order? Why is "overcast" the value 0, "rainy" the value 1, and "sunny" the value 2?

Label encoding is usually done by assigning consecutive values to an alphabetically sorted list of classes.

[7 / 15]

Alright, so now that every feature is numeric, we should be good to go, right?

Well, maybe.

It turns out that a lot of the machine learning algorithms are excellent at extracting subtle relationships in the data.

[8 / 15]

For example, if we are dealing with ratings, a 4-star movie is twice as good as a 2-star one.

Or, in the case of the temperature feature in our example, there's a clear relationship between the values: 80 is warmer than 79, and 75 is the coldest one.

[9 / 15]

Algorithms could pick up these relationships and exploit them!

What does this mean for our weather feature? Is "sunny" (value 2) twice as good as "rainy" (value 1)?

Of course not. We don't want the algorithm to "make up" an inexistent relationship.

[10 / 15]

If we don't want this to happen, we can't label encode the weather feature.

But there's something else we can do: we can one-hot encode it.

If that name seems gibberish to you, you aren't alone. Let's see how this works.

[11 / 15]

We know there are three possible classes for our weather features. One-hot encoding will turn that feature into three different binary new features:

▫️ weather_overcast
▫️ weather_rainy
▫️ weather_sunny

Each column will have a 0 or 1 value.

[12 / 15]

A different way to look at it is by thinking about the attached table.

We turned our weather feature into a vector with a value representing the specific class:

▫️ overcast: [1, 0, 0]
▫️ rainy: [0, 1, 0]
▫️ sunny: [0, 0, 1]

[13 / 15]

The advantage of this method is that we don't inadvertently introduce a relation between different values.

Our problems are not more! 😎

Here is the Python 🐍 code to get our new table with the new three weather features.

[14 / 15]

Here is the deal:

Writing these threads take a lot of time, but it's all worth it if you like/retweet them so others can benefit as well.

And while you are at it, follow me for more threads exploring the little things about machine learning.

Love ya! ✌️

[15 / 15]

🦕

Great question!

For high cardinality, I'd look into hashing encoding as a strong candidate.

There are many other options as well:

- Frequency Encoding
- Mean Encoding
- Weight/Evidence Encoding
- Probability Ratio Encoding
...

https://twitter.com/__solopiu/status/1364931572751888387?s=20

Too much of anything is unhealthy 😎

There is a price to pay for increasing the dimensionality of a dataset. You need to keep that in mind when deciding whether to use one-hot-encoding is the right choice.

https://twitter.com/jlcodes/status/1364948532839518208