Spurious features are a major issue for deep learning. Our new #NeurIPS2022 paper w/ @pol_kirichenko, @gruver_nate and @andrewgwils explores the representations trained on data with spurious features with many surprising findings, and SOTA results.
arxiv.org/abs/2210.11369
🧵1/6
arxiv.org/abs/2210.11369
🧵1/6
We use Deep Feature Reweighting (DFR) to evaluate feature representations: retrain the last layer of the model on group-balanced validation data. DFR worst group accuracy (WGA) tells us how much information about the core features is learned.
2/6
https://twitter.com/polkirichenko/status/1511883670923137026
2/6
While group robustness methods such as group DRO can improve WGA a lot, they don’t typically improve the features! With DFR, we recover the same performance for ERM and Group DRO. The improvement in these methods comes from the last layer, not features!
3/6
3/6
Better models learn the core feature better: in-distribution accuracy is linearly correlated with the DFR WGA. We don’t find qualitative differences between different types of architectures, such as CNNs and vision transformers: they all fall on the same line.
4/6
4/6
ImageNet pretraining (supervised or contrastive) has a major effect on the features, even on non-natural image datasets such as chest X-rays. With strong pretrained models, we achieve SOTA WGA on Waterbirds (97%) , CelebA (92%) and FMOW (50%) with ERM features.
5/6
5/6
Many more insights and results in the paper!
Paper: arxiv.org/abs/2210.11369
Code: github.com/izmailovpavel/…
NeurIPS event: neurips.cc/virtual/2022/p…
6/6
Paper: arxiv.org/abs/2210.11369
Code: github.com/izmailovpavel/…
NeurIPS event: neurips.cc/virtual/2022/p…
6/6
• • •
Missing some Tweet in this thread? You can try to
force a refresh
