Thread by @svpino

The two questions related to neural networks that I hear most often:

▫️ How many layers should I use?
▫️ How many neurons per layer should I use?

There are some rules of thumb that I'll share with you after you get your ☕️ ready.

🧵👇

First, let's get this out of the way:

A neural network with a single hidden layer can model any function regardless of how complex it is (assuming it has enough neurons.)

Check the "Universal Approximation Theorem" if you don't believe me.

↓

So, if we can do it all with a single layer, why bother adding more layers?

Well, it turns out that a neural network with a single layer will overfit really quick.

The more neurons you add to it, the better it will become at memorizing stuff.

That is bad news.

↓

Do you know one way to fight this overfitting? Adding more data. A ton of it.

Do you think you have that much data? I didn't think so.

Theoretically, yes, it's possible to learn everything with a single layer. Practically, a bad idea.

↓

So let's add more layers instead.

Deeper networks learn features at different levels of abstraction. This way, they can learn richer structures.

Let's take images of bunnies 🐇 and a network with 3 layers and see what may happen (keep reading...)

↓

Here is the hypothetical learning that could take place:

▫️ The 1st layer learns to recognize edges.

▫️ The 2nd layer learns shapes using the edges learned by the 1st layer.

▫️ The 3rd layer learns to recognize ears and noses using the shapes from the 2nd layer.

↓

In summary, more layers help a network generalize, as opposed to memorizing.

And there's more: because we have a more efficient structure at learning, we don't need a huge amount of neurons.

That's good news!

↓

Another small detail:

The more parameters we have, the slower it will be to train the network.

A deeper network will help us keep the number of parameters manageable.

↓

So going back to your original questions, here is a rule of thumb:

▫️ Prioritize deep and narrow networks over wide and shallow.

In other words: More layers with fewer neurons over fewer layers with more neurons.

But of course, it ultimately depends on your problem.

↓

If these threads help you find sanity among so many weights and biases, give me a follow and retweet this thing.

Do you know how many people are still asking the same questions? Let's settle it for them!

🦕

Parameters refer to the weight and biases of the network. The more neurons we have, the more parameters we will have to tune.

https://twitter.com/JurieHorneman/status/1367165977113354248?s=20

Imagine that we have 10 inputs, a hidden layer with 16 nodes (neurons), and an output layer with 3 nodes.

Here is how we compute the number of parameters:

Weights: 10 * 16 + 16 * 3 = 208
Biases: 16 + 3 = 19

Parameters: 208 + 19 = 227