Thread by @AtharvaIngle7

So, recently there was this question from @svpino that in theory, you can model any function using a neural network with a single hidden layer. However, deep networks are much more efficient than shallow ones. Why?
Here's a thread answering that question🧵

📌 Both shallow (network with one hidden layer) and deep networks(network with multiple hidden layers) are capable of approximating any function (theoretically). But, still, you get better results with deep neural networks. Here's why 👇

1. To match the performance of a multi-layered neural network you would require a large number of neurons in a single layer in the case of shallow networks.

2. In multi-layered neural networks, each layer can learn a different set of features. For example: if we are doing face recognition using CNN's, the lower layer learns about the edges and the learned edges can group together to detect parts of faces in the later layers.

3. Then, putting together different parts of faces (like eyes, nose, chin, etc) the network can combine them to detect different faces in the last layers

4. So, basically, the earlier layers are learning simpler functions and those learned functions are transferred to subsequent layers to learn more complicated functions.

5. The network is kind of creating abstraction in the intermediate layers of it. Depth gives us the possibility of abstracting relatively abstract concepts in the upper layers of the network.

6. Theoretically, to match the multi-layered neural network with a shallow network (with one hidden layer), you will require an exponentially large number of neurons, increasing the computational complexity of the model massively.

7. To summarize all of this, a deep network can fit the same function (which you are approximating) with fewer parameters than the shallow network

8. This tweet is heavily inspired by a lecture from Andrew Ng in his deeplearning.ai course. Here's the link to the video 👇