Thread by @volokuleshov

#Neurips2022 is now over---here is what I found exciting this year. Interesting trends include creative ML, diffusion models, language models, LLMs + RL, and some interesting theoretical work on conformal prediction, optimization, and more.

Two best paper awards went to work in creative ML---Imagen and LAION---in addition to many papers on improving generation quality, extending generation beyond images (e.g,. molecules), and more.

There was a lot of talk about ethics in creative ML (even an entire workshop on it), but I also saw fun applications in art, music & science (note: all workshops are recorded). Companies from Google to RunwayML had a big presence.

Diffusion models are another huge topic. Below is our DM circle 🙂 The panel at the DM workshop was great---key problems identified by panelists include discrete models, scalability, and going beyond Gaussian noising.

Several best paper awards went to diffusion model research, including Imagen, "elucidating the space of DMs", Riemannian score-based methods, and more.

Language models are obviously a big deal. Some papers reported interesting counter-intuitive phenomena (see pic), others reported interesting connections to RL. Bonus: best paper award for Chinchilla

LLM+RL is getting a lot of attention. Phil Blunsom gave a great workshop talk on interpreting in-context learning as adaptive computation. Some best papers in bigRL---MineDojo and Procthor.

ChatGPT also happened 🤯

I also found there was a lot of interesting theory work. Emmanuel Candes' keynote was on conformal prediction---a field that really blew up in the last 2-3 years. Turns out you can get confidence intervals in ML on non-IID data.

Also, lots of interesting theory on optimization, SGD, including two best paper awards. Learned optimizers might be making a comeback too!

My predictions for next year: lots of new extensions of diffusion models (discreteness, new types of diffusions). I also think LLMs will soon be smaller and easier to use. I'm excited for 2023!