Thread by @DeepMind

Multimodal transformers achieve impressive results on many tasks like Visual Question Answering and Image Retrieval, but what contributes most to their success? dpmd.ai/3h8u23Z (1/)

This work explores how different architecture variations, pretraining datasets, and losses impact multimodal transformers’ performance on image retrieval: dpmd.ai/3eENAtF

(By Lisa Anne Hendricks, John Mellor, Rosalia Schneider, @jalayrac & @aidanematzadeh) (2/)

Multimodal transformers outperform simpler dual encoder architectures when the amount of data is held constant. Interestingly, larger datasets don’t always improve performance. (3/)

This work shows that language similarity between between pretraining and task and dataset noise are important. Also, the masked region modelling loss and a contrastive image-text matching loss do not contribute to models’ performance. (4/4)