Check out BatchEnsemble: Efficient Ensembling with Rank 1 Perturbations at the #NeurIPS2019 Bayesian DL workshop. Better accuracies and uncertainty than dropout and competitive with ensembles across a wide range of tasks. 1/- 
It’s a drop in replacement for individual layers, like dropout, batchnorm, and variational layers and is available with baselines and Bayesian layers at github.com/google/edward2. 2/-
Unlike ensembles, it’s trained end to end under a single loss function (NLL) and computation can be parallelized across ensemble members in GPU/TPUs. BatchEnsemble is like a new parameterization for neural nets. 3/-
On ResNet-50, it’s only 0.2% more parameters than vanilla. Ensembles over even small sizes like 4 have 4x the memory cost and 4 separate forward passes at test time. (Similarly, typical VI is 2x and underfits in terms of accuracy.) 4/-
Training wise, we use 300 epochs compared to vanilla SGD with momentum which uses 200 epochs and make few differences in optimizer/LR. This is comparable to VI; SGMCMC takes quite a bit longer to get diverse samples (each of which are a copy of the full set of parameters). 6/-
I think there’s something fundamental about functional diversity by aggregating multimodal solutions, and on low rank subspaces so you can make even bigger models to be Bayesian about. This builds on lots of excellent work in the community. Chat with us at the poster! 6/-
