,
5 tweets,
1 min read
Read on Twitter
"Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit" with Belinda Tzen arxiv.org/abs/1905.09883 1/5
Neural SDEs are simply deep latent Gaussian models in the limit of infinitely many layers, with appropriate scaling of the layer-to-layer noise. In our earlier work (to appear in COLT 2019), we have analyzed the expressiveness of such diffusion-based generative models. 2/5
In this follow-up work, we use ideas from Ito calculus to develop a variational inference framework for neural SDEs. In particular, the latent space is now the Wiener space of continuous paths and the primitive random object is the standard Wiener process. 3/5
Girsanov's theorem gives us a natural Wiener-space counterpart of the reparametrization trick, and the mean-field approximation takes a particularly clean form --- it is simply the Wiener process with a deterministic non-stationary drift. 4/5
Finally, we use some results on stochastic flows to show that the solutions of neural SDEs can be differentiated w.r.t. the network parameters, and one can compute these pathwise derivatives using any black-box SDE solver and automatic differentiation. 5/5
