A good way to understand how shit works is by breaking it down as much as you can.
Here is some code showing Dropout working on an array. And this is a thread explaining how it works.
☕️🧵👇
Here is some code showing Dropout working on an array. And this is a thread explaining how it works.
☕️🧵👇
First, the code.
I want you to notice that Dropout does a couple of things:
▫️ It zeroes-out a percentage of the units.
▫️ It scales the remaining units to account for the missing values.
The second one wasn't obvious to me.
I want you to notice that Dropout does a couple of things:
▫️ It zeroes-out a percentage of the units.
▫️ It scales the remaining units to account for the missing values.
The second one wasn't obvious to me.
Remember those kids from school that sat together and copied from each other during exams?
They aced every test but were hardly brilliant, remember?
Eventually, the teacher had to set them apart. That was the only way to force them to learn.
They aced every test but were hardly brilliant, remember?
Eventually, the teacher had to set them apart. That was the only way to force them to learn.
The same happens with neural networks.
Sometimes, a few hidden nodes create associations that do most of the work, forcing the network to ignore the rest.
This is called co-adaptation, and it prevents networks from generalizing.
Sometimes, a few hidden nodes create associations that do most of the work, forcing the network to ignore the rest.
This is called co-adaptation, and it prevents networks from generalizing.
We can solve this problem like teachers do: breaking the associations that prevent the network from learning.
This is what Dropout is for.
During training, Dropout randomly removes some of the units.
This forces the network to learn in a balanced way.
This is what Dropout is for.
During training, Dropout randomly removes some of the units.
This forces the network to learn in a balanced way.
Now every unit is on its own and can't rely on other units to do their work. They have to work harder by themselves.
Dropout works very well, and it's one of the main mechanisms to reduce overfitting.
Dropout works very well, and it's one of the main mechanisms to reduce overfitting.
Great question!
The remaining nodes will pick up the slack and learn.
If we keep dropping nodes, the ones remaining need to work harder and independently to learn.
The remaining nodes will pick up the slack and learn.
If we keep dropping nodes, the ones remaining need to work harder and independently to learn.
https://twitter.com/ryanhamisugly/status/1371115402508476419
This is a very good question!
Even if you start with a narrower network, there's no guarantee that nodes won't start collaborating and rendering useless the rest of the nodes. Dropout will ensure that doesn't happen.
Even if you start with a narrower network, there's no guarantee that nodes won't start collaborating and rendering useless the rest of the nodes. Dropout will ensure that doesn't happen.
https://twitter.com/Nomad111Fly/status/1371127471005831179?s=20
• • •
Missing some Tweet in this thread? You can try to
force a refresh
