Thread by @jaschasd

CALL FOR TASKS CAPTURING LIMITATIONS OF LARGE LANGUAGE MODELS

We are soliciting contributions of tasks to a *collaborative* benchmark designed to measure and extrapolate the capabilities and limitations of large language models. Submit tasks at github.com/google/BIG-Ben…
#BIGbench

All accepted task submitters will be co-authors on the paper releasing the benchmark. Teams at Google and OpenAI will further evaluate BIG-Bench on their best-performing model architectures, across models spanning from tens of thousands through hundreds of billions of parameters.

We encourage submission of tasks by researchers in fields other than computer science which probe the nature of language or intelligence, including: linguistics, cognitive science, philosophy, neuroscience, psychology, animal intelligence, and logic.

We also encourage submission of tasks which quantify social bias in language models. Including measures of bias in a standard language model benchmark will motivate future research countering it.

The benchmark and results of the model evaluation will be released at the ICLR 2021 Workshop on Enormous Language Models.

See github.com/google/BIG-Ben… for more details.