Thread by @balajiln

Why do deep ensembles trained with just random initialization work surprisingly well in practice? 

In our recent paper arxiv.org/abs/1912.02757 with @stanislavfort & Huiyi Hu, we investigate this by using insights from recent work on loss landscape of neural nets. 

More below:

@stanislavfort 2) One hypothesis is that ensembles may lead to different modes while scalable Bayesian methods may sample from a single mode.

We measure the similarity of function (both in weight space and function space) to test this hypothesis.

@stanislavfort 3) t-SNE plot of predictions along training trajectories (marked by different colors) shows that random initialization leads to diverse functions. Sampling functions from a subspace corresponding to a single trajectory increases diversity but not as much as random init.

@stanislavfort 4) From a bias-variance perspective, we care about both accurate solutions (low bias) and diverse solutions (as decorrelation reduces variance).  

Given a reference solution, we plot diversity vs accuracy to measure how different methods trade-off diversity vs accuracy.

@stanislavfort 5) We also validate the hypothesis by building low-loss tunnels between solutions found by different random inits. While points along low loss tunnel have similar accuracies, the function space disagreement between them & the two end points shows that the modes are diverse.

@stanislavfort If you'd like to learn more, check out our paper arxiv.org/abs/1912.02757 :)

@stanislavfort will also be giving a contributed talk about our work on Dec 13 (Friday) 9-915 AM and presenting a poster at the Bayesian deep learning workshop (bayesiandeeplearning.org) at #NeurIPS2019