Thread by @stevebach

If you’re at #ICLR2022, hope you’ll check out our spotlighted poster: “Multitask Prompted Training Enables Zero-Shot Task Generalization.” arxiv.org/abs/2110.08207

Poster session 5, Tue 1:30-3:30 ET

This work was a big undertaking from many at the @BigscienceW Workshop, particularly @SanhEstPasMoi, @albertwebson, @colinraffel, and @srush_nlp. It’s been awesome to see all the people already using and building on the T0 family of models for zero-shot learning.

There’s rightly been a lot of excitement around the zero-shot performance of T0 and similar, concurrent approaches like FLAN (ai.googleblog.com/2021/10/introd…).

I also want to highlight the data-centric side of the T0 work.

The hypothesis behind T0 was that we could induce better zero-shot behavior by fine-tuning language models on examples of prompted tasks. But what prompts should we use? Our answer to this question was to iteratively build and refine a collection of over 2000 prompt templates.

We focused on trying to encourage robustness to prompt formulations. Our team developed multiple prompts for 170+ datasets. When applied to the corresponding datasets, this approach can create 100 million+ prompted examples, a subset of which were used to train T0.

Experiments in the paper show that training on multiple prompted formats for the same dataset indeed increased robustness.

The takeaway: engineering the training data for T0 was a key part of getting it to work. Pilot experiments surfaced issues with our preliminary prompts. Along the way we developed guidelines to curate a diverse set of quality prompts.

T0 and PromptSource are just pieces of this emerging trend. For example, check out the awesome work on Natural Instructions led by @Swarooprm7, @yizhongwyz, and @DanielKhashabi. v1 was a precursor to PromptSource, and their new v2 is a massive set of detailed task instructions.

See also @srush_nlp’s great perspective:

https://twitter.com/srush_nlp/status/1450539883081179139

Access T0++ here: huggingface.co/bigscience/T0pp
And PromptSource: github.com/bigscience-wor…