Discover and read the best of Twitter Threads about #ICRL2020
Most recents (1)
Very happy to share our latest work accepted at #ICRL2020: we prove that a Self-Attention layer can express any CNN layer. 1/5
📄Paper: openreview.net/pdf?id=HJlnC1r…
🍿Interactive website : epfml.github.io/attention-cnn/
🖥Code: github.com/epfml/attentio…
📝Blog: jbcordonnier.com/posts/attentio…
📄Paper: openreview.net/pdf?id=HJlnC1r…
🍿Interactive website : epfml.github.io/attention-cnn/
🖥Code: github.com/epfml/attentio…
📝Blog: jbcordonnier.com/posts/attentio…
Two *necessary* conditions (often met in practice):
(a) Multiple heads, ex: 3x3 kernel requires 9 heads,
(b) Relative positional encoding to allow translation invariance.
Each head can attend on pixels at a fixed shift from the query pixel forming the pixel receptive field. 2/5
(a) Multiple heads, ex: 3x3 kernel requires 9 heads,
(b) Relative positional encoding to allow translation invariance.
Each head can attend on pixels at a fixed shift from the query pixel forming the pixel receptive field. 2/5
Our work explains the recent success of Transformer architecture applied to vision:
Attention Augmented Convolutional Networks. @IrwanBello et al., 2019. arxiv.org/abs/1904.09925
Stand-Alone Self-Attention in Vision Models. Ramachandran et al., 2019. arxiv.org/abs/1906.05909
3/5
Attention Augmented Convolutional Networks. @IrwanBello et al., 2019. arxiv.org/abs/1904.09925
Stand-Alone Self-Attention in Vision Models. Ramachandran et al., 2019. arxiv.org/abs/1906.05909
3/5
