Thread by @TheSequenceAI

Pre-trained language models have been one of the most important breakthroughs in the recent years of deep learning.

What models are used in super large-scale language tasks?
Thread👇

Pre-trained language models are trained in massive text datasets.

Thanks to transformer architectures, we can implement pre-trained language models adapted to specific tasks. For example, question-answering or language modeling.
2/⬇️

Transformers opened the door to a new era of innovation in NLU. And the attention mechanism used in transformers = one of the most impactful developments in the last years of ML.

More about transformers & attention here: thesequence.substack.com/p/thesequence-…
3/⬇️

Researches from @MSFTResearch moved further. They introduced one of the first generative models that could be used in super large-scale language tasks. It was @ChunyuanLi @icaruszyz @JianfengGao0217 Xiang Gao, Yuan Li, Xiujun Li, Baolin Peng.
4/⬇️

They called their model Optimus.

Optimus attempts to combine large pre-trained language models (language understanding) and generation tasks in a very clever architecture using generative models.
5/⬇️

The Optimus architecture includes a BERT-based encoder and a GPT-2-based decoder.

From that perspective, Optimus = variational auto-encoder (VAE) architecture that combines encoder and decoder layers.
6/⬇️

The full paper about Optimus is called "Optimus: Organizing sentences via pre-trained modeling of a latent space".

You can find it here: chunyuan.li/papers/Optimus…
And the code with data: github.com/ChunyuanLI/Opt…
7/⬇️

If you'd like to find a bite-sized, highly concentrated overview of this paper click the link below. You'll move to TheSequence Edge#7, our educational newsletter.
thesequence.substack.com/p/edge7
8/8