Thread by @jb_cordonnier

Very happy to share our latest work accepted at #ICRL2020: we prove that a Self-Attention layer can express any CNN layer. 1/5

📄Paper: openreview.net/pdf?id=HJlnC1r…
🍿Interactive website : epfml.github.io/attention-cnn/
🖥Code: github.com/epfml/attentio…
📝Blog: jbcordonnier.com/posts/attentio…

Two *necessary* conditions (often met in practice):
(a) Multiple heads, ex: 3x3 kernel requires 9 heads,
(b) Relative positional encoding to allow translation invariance.

Each head can attend on pixels at a fixed shift from the query pixel forming the pixel receptive field. 2/5

Our work explains the recent success of Transformer architecture applied to vision:

Attention Augmented Convolutional Networks. @IrwanBello et al., 2019. arxiv.org/abs/1904.09925

Stand-Alone Self-Attention in Vision Models. Ramachandran et al., 2019. arxiv.org/abs/1906.05909

3/5

Interactive website displays attention maps:
- some heads ignore content and attend on pixels at *fixed* shifts (confirms theory) sliding a grid-like receptive field,
- some heads seem to use content-based attention -> expressive advantage over CNN.
🍿 epfml.github.io/attention-cnn 4/5

I deeply appreciated the engagement of the reviewers (esp. the critics from Reviewer 3). I am thankful to @loukasa_tweet and Martin Jaggi for their support at @epfl_en. My Ph.D. is funded by @SDSCdatascience, Andreas is supported by @snsf_ch. Adis Ababa, here we come! 🇪🇹🎉 5/5