Thread by @sorenmind

Tired of waiting 💤 while your model trains? Try skipping points that are already learned, not learnable or not worth learning! Robustly reduces required training steps 🏎 by >10x ! to reach the same accuracy on big web-scraped data

📜ICML 2022 paper: arxiv.org/abs/2206.07137

Training on big web-scraped data can take ages 💤 But lots of compute and time is wasted on redundant and noisy points that are already learned, not learnable, or not even worth learning.

What if we just skip these points? Our method—RHO-LOSS—trains in far fewer gradient steps than prior art, boosts accuracy, and speeds up training on 8 datasets, lots of hyperparameters, and 10 architectures (MLPs, CNNs, and BERT).

Existing ideas, like skipping points with low loss, accidentally prioritise noisy and less relevant which are common in real-world data and web-scrapes, but barely help generalization.

That’s where Reducible Holdout Loss Selection (RHO-LOSS) comes in. We select points that most reduce the generalisation loss and show that this objective has a simple and cheap but close approximation (line 7).

This gives an intuitive result: in a precise sense, the optimal points for fast training are learnable, worth learning, and not yet learned.

📜#icml2022 paper: arxiv.org/pdf/2206.07137…

Check out the paper for:
✨ re-using a single small auxiliary model to accelerate training across multiple architectures
✨ RHO-LOSS as an efficient approximation to optimal selection, derived in the language of probabilistic modelling
✨ why this works so well 😊

Work done at @oatml @CohereAI with great collaborators @JanMBrauner, @MrinankSharma @mtrazzak, @BlackHC, Winnie Xu, Ben Höltgen, @adrien_morisot, @aidangomezzz, @seb_far, @yaringal

⌨️Code: github.com/OATML/RHO-Loss

Correction to the lap of honour: @OATML_Oxford is where it happened 😊

Might be of interest to @yieldthought (thanks for your kind words on the workshop paper!)