,
4 tweets,
3 min read
Read on Twitter
New blog post: How does batch norm _really_ help optimisation?
We go on a tour of bad inits, degenerate networks and spiky Hessians - all in a Colab notebook:
colab.research.google.com/github/davidcp…
Summary 👇
We go on a tour of bad inits, degenerate networks and spiky Hessians - all in a Colab notebook:
colab.research.google.com/github/davidcp…
Summary 👇
1/ Early signs of trouble.
We learn that deep ReLU nets with He-init, but no batch norm, basically ignore their inputs! (Check out arxiv.org/abs/1902.04942 by Luther, @SebastianSeung for background.)
Easy to miss if you pool across channels:
We learn that deep ReLU nets with He-init, but no batch norm, basically ignore their inputs! (Check out arxiv.org/abs/1902.04942 by Luther, @SebastianSeung for background.)
Easy to miss if you pool across channels:
2/ How to break your ResNet.
Next we go looking for trouble: we find degenerate networks near the training trajectory by (mis)using backprop. Batch norm comes to the rescue!
Next we go looking for trouble: we find degenerate networks near the training trajectory by (mis)using backprop. Batch norm comes to the rescue!
3/ Spikes.
Finally we connect to instability of SGD and outlying eigenvalues of the Hessian (found by @leventsagun, Bottou, @ylecun.)
The mystery of the spiky Hessian is resolved along with the secrets of batch norm!
Finally we connect to instability of SGD and outlying eigenvalues of the Hessian (found by @leventsagun, Bottou, @ylecun.)
The mystery of the spiky Hessian is resolved along with the secrets of batch norm!
