Thread by @svpino

Do you wanna know why do we use ReLU when doing deep learning?

When starting out with neural networks, it's common to work with examples using the sigmoid activation function.

The sigmoid function squeezes any input value to a value between 0 and 1.

This is a 🧵👇

There's a problem with the sigmoid function: it saturates quickly.

This means that smaller and larger values will get concentrated around 0 and 1, respectively.

The function is only sensitive to values around the midpoint.

(2 of 5)

Once saturated, the weights will stop changing, and the network will not learn anything useful.

If your network is not too deep, this will not be an issue. But if you have a buttload of layers, you'll likely run into the problem.

This sucks.

(3 of 5)

ReLU doesn't have the same issue.

ReLU turns any negative value into 0 and leaves the rest untouched.

It has some cool properties:

▫️ Easy to implement (Just a max)
▫️ Linear properties (for values greater than 0.0)
▫️ No vanishing gradient problem

(4 of 5)

With ReLU, we get the benefits of linear functions and non-linearities (for values under 0.0) to learn interesting stuff.

ReLU is one of those things that made deep learning possible.

(5 of 5)

In practice, you can control the dying ReLU problem by carefully selecting the appropriate learning rate.

Basically, set the learning rate too high, and half of your network may die. But with the appropriate value, dying ReLU will not be a problem.

https://twitter.com/erwinkhn/status/1374105164181868555?s=20