Thread by @colinraffel

The #ICLR2021 Workshop on Enormous Language Models (WELM) is tomorrow, May 7th!

Full info: welmworkshop.github.io
Livestream: welmworkshop.github.io/livestream/
gathertown info for ICLR registrants: iclr.cc/virtual/2021/w…

Thread summarizing the talks & panels ⬇️ (1/14)

Our first talk will be by Thomas Margoni, who will provide some legal perspective on the use of web data for training large language models. He'll touch on topics like copyright law, rights, and licenses, as they pertain to training data for LMs. (2/14)

Then, @JesseDodge will give a talk on how to document datasets and improve reproducibility of research. He'll discuss the NLP reproducibility checklist, a recent study on documenting C4, and a framework for modeling bias in data. (3/14)

We'll then have @emilymbender and @mcmillan_majora present the recent (but already seminal) Stochastic Parrots 🦜 paper, which places a critical lens on enormous LMs from many perspectives, including environmental and financial costs and dataset accountability. (4/14)

After a short break, @Thom_Wolf will give an introduction to the nascent Big Science effort, whose goal is to build a giant LM through a massive community collaboration (taking inspiration from similar efforts in particle physics). (5/14)

Then, @em_dinan will discuss her extensive work on measuring, documenting, and mitigating toxic generation from large LMs. After arguing why this is an important and challenging problem, she'll discuss adversarial human-in-the loop benchmarking as a path for progress. (6/14)

Our first panel of the day will be on “Bias, safety, copyright, and efficiency”, moderated by @katherine1ee and will include panelists @Thom_Wolf, Thomas Margoni, @em_dinan, @natschluter, and @JesseDodge. (7/14)

As part of WELM we created the participant-led BIG-bench (github.com/google/BIG-ben…) for more rigorous evaluation of giant LMs. After an introduction and overview by @jaschasd, we'll have 10 spotlights and talks by @megamor2, @oliverrr_shen, Nathan A. Chi, and Rowan Jacobs. (8/14)

After the BIG-Bench session, Noam Shazeer will give a talk on a new way to characterize and describe the distributed computation algorithms required for training and serving enormous language models. (9/14)

Then, @ml_perception will discuss various architectures that can dramatically expand language modeling capabilities without using brute-force scaling, including nonparametric and sparsely-activated models. (10/14)

After that, Nicholas Carlini will discuss work that demonstrates the possibility of extracting training data from large language models -- even in the case where the model has not "overfit" to the training data in the traditional sense. (11/14)

We'll then have a talk by @AlisonGopnik describing how young children can perform causal, counterfactual, and relational inference, and what this means for building language models that should have similar capabilities. (12/14)

Our last talk will be by @yejinchoinka, a professor at UW who has led the development of many large LMs and benchmarks from within academia. She'll lend perspective as to how to do work in this space without access to industry-sized resources. (13/14)

Finally, our workshop will end with a second panel on “Extrapolating the capabilities of language models” with @AlisonGopnik, @yejinchoinka, @ml_perception, and @emilymbender. I am incredibly excited about this workshop and hope to see you (virtually) there! (14/14)