Profile picture
, 47 tweets, 35 min read
My Authors
Read all threads
With @iclr_conf #ICLR2020 over and a bit of sleep under my belt, I'd like to give my short summary of a truly great event---and offer a list of the papers I enjoyed seeing (for those who are into that kind of thing).
In general, I feel lucky to live in a time where we have venues like these full of really interesting papers on the intersection between NLP and ML (and others, but that's what I personally am most into, so my experience is biased).
First off, echoing what everyone else concluded: the website was great. For those who didn't attend, I hope you'll get to see it soon. Having a prerecorded 5-minute talk for each paper along with the slides you could click through made for excellent paper browsing in my mind:
I just clicked on each title that sounded potentially interesting, clicked through the slides, watched the talk, and attended the Zoom Q&A to ask a question or two. I honestly wish we could keep the prerecorded talks and online slides for in-person conferences!
The RocketChat instance worked really well, too, and allowed for nice conference-global communication. The socials were interesting and diverse, some structured more like workshops and others more in the spirit of a social gathering with chat.
I especially enjoyed the Language social, the lingustics vs. NLP room was great fun! Keynotes were interesting and the fact that they were also mostly prerecorded meant easy scheduling. And hoping I don't forget any other thoughts I have, here's my list of fun papers:
"Robustness Verification for Transformers"

Zhouxing Shi (@zhouxingshi), Huan Zhang, Kai-Wei Chang (@kaiwei_chang), Minlie Huan, Cho-Jui Hsieh
"Picking Winning Tickets Before Training by Preserving Gradient Flow"

Chaoqi Wang, Guodong Zhang (@guodzh), Roger Grosse (@RogerGrosse)
"Understanding the Limitations of Variational Mutual Information Estimators"

Jiaming Song (@baaadas), Stefano Ermon (@StefanoErmon)
openreview.net/forum?id=B1x62…
"Generalization through Memorization: Nearest Neighbor Language Models"

Urvashi Khandelwal (@ukhndlwl), Omer Levy (@omerlevy_), Dan Jurafsky (@jurafsky), Luke Zettlemoyer (@LukeZettlemoyer), Mike Lewis (@ml_perception)
"Your classifier is secretly an energy based model and you should treat it like one"

Will Grathwohl (@wgrathwohl), KC Wang (@kcjacksonwang), Joern-Henrik Jacobsen (@jh_jacobsen), @DavidDuvenaud, Mohammad Norouzi (@Mo_Norouzi), Kevin Swersky (@kswersk)
"Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework"

Zirui Wang, Jiateng Xie, Ruochen Xu, Yiming Yang, Graham Neubig (@gneubig), Jaime G. Carbonell
"Gradient-Based Neural DAG Learning"

Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu (@TristanDeleu), Simon Lacoste-Julien (@SimonLacosteJ)
"Plug and Play Language Models"

Sumanth Dathathri (@sdathath), Andrea Madotto (@AndreaMadotto), Janice Lan (@lanjanice), Jane Hung (@jhung0), Eric Frank (@IHaveSweaters), Piero Molino (@w4nderlus7), Jason Yosinski (@jasonyo), Rosanne Liu (@savvyRL)
"ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators"

Kevin Clark (@clark_kev), Minh-Thang Luong (@lmthang), Quoc V. Le (@quocleix), Christopher D. Manning (@chrmanning)
"A Probabilistic Formulation of Unsupervised Text Style Transfer"

Junxian He (@junxian_he), Xinyi Wang (@cindyxinyiwang), Graham Neubig (@gneubig), Taylor Berg-Kirkpatrick (@BergKirkpatrick)
"Estimating Gradients for Discrete Random Variables by Sampling without Replacement"

Wouter Kool (@wouterkool), Herke van Hoof, Max Welling (@wellingmax)
openreview.net/forum?id=rklEj…
"Functional vs. parametric equivalence of ReLU networks"

Mary Phuong, Christoph H. Lampert
openreview.net/forum?id=Bylx-…
"On Identifiability in Transformers"

Gino Brunner (@ginozkz), Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, Roger Wattenhofer
openreview.net/forum?id=BJg1f…
"Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP"

Haonan Yu, Sergey Edunov, Yuandong Tian (@tydsh), Ari S. Morcos (@arimorcos)
openreview.net/forum?id=S1xnX…
"Neural Tangents: Fast and Easy Infinite Neural Networks in Python"

Roman Novak, Lechao Xiao (@Locchiu), Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, Samuel S. Schoenholz (@sschoenholz)
"SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models"

Yucen Luo (@MarianLuo), Alex Beatson (@AlexBeatson), Mohammad Norouzi (@mo_norouzi), Jun Zhu (@junyanz89), @DavidDuvenaud, Ryan P. Adams, Ricky T. Q. Chen (@RickyTQChen)
openreview.net/forum?id=SylkY…
"Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning"

Ruqi Zhang, Chunyuan Li (@ChunyuanLi), Jianyi Zhang, Changyou Chen, Andrew Gordon Wilson (@andrewgwils)
"The Curious Case of Neural Text Degeneration"

Ari Holtzman (@universeinanegg), Jan Buys, Li Du, Maxwell Forbes, Yejin Choi (@YejinChoinka)
openreview.net/forum?id=rygGQ…
"Thieves on Sesame Street! Model Extraction of BERT-based APIs"

Kalpesh Krishna (@kalpeshk2011), Gaurav Singh Tomar (@gst1988), Ankur P. Parikh (@ank_parikh), Nicolas Papernot (@NicolasPapernot), Mohit Iyyer (@MohitIyyer)
"Improving Neural Language Generation with Spectrum Control"

Lingxiao Wang, Jing Huang, Kevin Huang, Ziniu Hu, Guangtao Wang, Quanquan Gu (@QuanquanGu)
openreview.net/forum?id=ByxY8…
"Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks"

Sanjeev Arora (@prfsanjeevarora), Simon S. Du, Zhiyuan Li, Ruslan Salakhutdinov (@rsalakhu), Ruosong Wang (@RuosongW), Dingli Yu (@dingli_yu)
"BERTScore: Evaluating Text Generation with BERT"

Tianyi Zhang (@Tianyi_Zh), Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi (@yoavartzi)
"Are Transformers universal approximators of sequence-to-sequence functions?"

Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, Sanjiv Kumar
openreview.net/forum?id=ByxRM…
"Finite Depth and Width Corrections to the Neural Tangent Kernel"

Boris Hanin (@BorisHanin), Mihai Nica
openreview.net/forum?id=SJgnd…
"A Mutual Information Maximization Perspective of Language Representation Learning"

Lingpeng Kong (@ikekong), Cyprien de Masson d'Autume, Lei Yu, Wang Ling, Zihang Dai (@ZihangDai), Dani Yogatama (@DaniYogatama)
"Reformer: The Efficient Transformer"
Nikita Kitaev, Lukasz Kaiser (@lukaszkaiser), Anselm Levskaya (@anselmlevskaya)
"A Theory of Usable Information under Computational Constraints"

Yilun Xu, Shengjia Zhao (@shengjia_zhao), Jiaming Song (@baaadas), Russell Stewart, Stefano Ermon (@StefanoErmon)
openreview.net/forum?id=r1eBe…
"Encoding word order in complex embeddings"

Benyou Wang (@wabyking), Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, Jakob Grue Simonsen
openreview.net/forum?id=Hke-W…
"Mogrifier LSTM"

Gábor Melis (@GaborMelis), Tomáš Kočiský (@TomasKocisky), Phil Blunsom
"Learning The Difference That Makes A Difference With Counterfactually-Augmented Data"

Divyansh Kaushik (@dkaushik96), Eduard Hovy, Zachary Lipton (@zacharylipton)
"Cross-Lingual Ability of Multilingual BERT: An Empirical Study"

Karthikeyan K, Zihan Wang, Stephen Mayhew (@mayhewsw), Dan Roth
openreview.net/forum?id=HJeT3…
"A Constructive Prediction of the Generalization Error Across Scales"

Jonathan S. Rosenfeld (@jonsrosenfeld), Amir Rosenfeld (@AmirRosenfeld), Yonatan Belinkov (@boknilev), Nir Shavit
"Neural Module Networks for Reasoning over Text"

Nitish Gupta (@nitish_gup), Kevin Lin (@nlpkevinl), Dan Roth (@dannydanr), Sameer Singh (@sameer_), Matt Gardner (@nlpmattg)
"ALBERT: A Lite BERT for Self-supervised Learning of Language Representations"

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut
openreview.net/forum?id=H1eA7…
"Extreme Classification via Adversarial Softmax Approximation"

Robert Bamler, Stephan Mandt (@s_mandt)
openreview.net/forum?id=rJxe3…
"A Latent Morphology Model for Open-Vocabulary Neural Machine Translation"

Duygu Ataman (@_dataman_), Wilker Aziz (@wilkeraziz), Alexandra Birch (@alexandrabirch1)
"Unsupervised Model Selection for Variational Disentangled Representation Learning"

Sunny Duan, Loic Matthey (@Azhag), Andre Saraiva (@andresnds), Nick Watters, Chris Burgess (@cpburgess_), Alexander Lerchner (@AlexLerchner), Irina Higgins
openreview.net/forum?id=SyxL2…
"From Variational to Deterministic Autoencoders"

Partha Ghosh (@ParthaG64920039), Mehdi S. M. Sajjadi, Antonio Vergari (@tetraduzione), Michael Black (@Michael_J_Black), Bernhard Scholkopf (@bschoelkopf)
"Language GANs Falling Short"

Massimo Caccia (@MassCaccia), Lucas Caccia (@LucasPCaccia), William Fedus (@LiamFedus), Hugo Larochelle (@hugo_larochelle), Joelle Pineau, Laurent Charlin (@lcharlin)
openreview.net/forum?id=BJgza…
"Multilingual Alignment of Contextual Word Representations"

Steven Cao, Nikita Kitaev, Dan Klein
openreview.net/forum?id=r1xCM…
Thanks again to all @iclr_conf #ICLR2020 organizers and volunteers for an amazing event exceeding all expectation, to all authors for answering my naive questions, and to all participants for all the interesting discussions you made possible! :) See you next time!
Missing some Tweet in this thread? You can try to force a refresh.

Enjoying this thread?

Keep Current with Sabrina J. Mielke

Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Related hashtags

#ICLR2020

Embed code for your website

×
Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!