Very happy to share our latest work accepted at #ICRL2020: we prove that a Self-Attention layer can express any CNN layer. 1/5

📄Paper: openreview.net/pdf?id=HJlnC1r…
🍿Interactive website : epfml.github.io/attention-cnn/
🖥Code: github.com/epfml/attentio…
📝Blog: jbcordonnier.com/posts/attentio… Two *necessary* conditions (often met in practice):
(a) Multiple heads, ex: 3x3 kernel requires 9 heads,
(b) Relative positional encoding to allow translation invariance.

Each head can attend on pixels at a fixed shift from the query pixel forming the pixel receptive field. 2/5