Thread by @svpino

Why is it important to introduce non-linearities in a neural network?

The short answer: So we can solve more interesting problems.

The left image shows a classification problem that can be solved using a single dividing line. The image on the right is much more complex.

🧵👇

Here is a neural network with 2 hidden layers of 4 neurons. The activation is set to "Linear."

In just a few epochs, the network finds the correct solution.

Notice how the network uses a single dividing line in the output. That's all it can do with linear activations.

👇

If we try the same network on the more complex problem, it will struggle to classify the data correctly.

We haven't introduced non-linearities in this network, so it won't find the proper solution for this type of problem.

👇

Let's now change the activation to ReLU. This will introduce the non-linearity we need for the network to become more powerful.

Same exact network structure, with just a different activation, and we are now in business!

👇

A neural network with linear activation functions and 𝑛 layers with 𝑚 hidden units is equivalent to a linear neural network without hidden layers.

In other words, a neural network without non-linear activation functions can only find linear separation boundaries.

👇

The pictures on this thread come from the TensorFlow Playground.

If you want to see how a neural network in action and analyze how it works, take a look at it!

playground.tensorflow.org

👇

This thread is answering the first question from the set of machine learning questions that you can find attached.

Stay tuned for the rest of the answers and help me spread more machine learning content to more people in the community!

https://twitter.com/svpino/status/1350678998213222400?s=20