Thread by @JayAlammar

On the transformer side of #acl2020nlp, three works stood out to me as relevant if you've followed the Illustrated Transformer/BERT series on my blog:
1- SpanBERT
2- BART
3- Quantifying Attention Flow
(1/n)

SpanBERT (by @mandarjoshi_ @danqi_chen @YinhanL @dsweld @LukeZettlemoyer @omerlevy_) came out last year but was published in this year's ACL. It found that BERT pre-training is better when you mask continuous strings of tokens, rather than BERT's 15% scattered tokens.

BART (@ml_perception @YinhanL @gh_marjan @omerlevy_ @vesko_st @LukeZettlemoyer) presents a way to use what we've learned from BERT (and spanBERT) back into encoder-decoder models, which are especially important for summarization, machine translation, and chatbots. 3/n #acl2020nlp

In "Quantifying Attention Flow", @samiraabnar and Willem Zuidema show that in higher/later transformer blocks, you shouldn't rely on raw attention weights to tell which tokens are being attended to.

aclweb.org/anthology/2020…

4/n #acl2020nlp

(I now strongly prefer visualizing transformers with input coming from the top to the bottom, as that reads more naturally as one is scrolling down a page. But we're kinda stuck with language like "top layer" referring to the last layer)

SpanBERT paper from TACL (thank you @danqi_chen!) :
mitpressjournals.org/doi/pdf/10.116…

BART paper:
aclweb.org/anthology/2020…