Share this page!

Enter URL or ID to Unroll

You can paste full URL like: https://x.com/threadreaderapp/status/1644127596119195649
or just the ID like: 1644127596119195649

How to get URL link on X (Twitter) App

  1. On the Twitter thread, click on or icon on the bottom
  2. Click again on or Share Via icon
  3. Click on Copy Link to Tweet
  4. Paste it above and click "Unroll Thread"!
  5. More info at Twitter Help
Andrew Brock Profile picture
Andrew Brock
@ajmooch
Dimensionality Diabolist, Affine combination of Mad Scientist and Village Idiot
Feb 12, 2021 • 7 tweets • 3 min read
Our most recent work on training Normalizer-Free nets!

We focus on developing performant architectures which train fast, and show that a simple technique (Adaptive Grad Clipping, or AGC) allows us to train with large batches and heavy augmentations and reach state-of-the-art. AGC is based on the idea that the magnitude of a gradient shouldn't be too large relative to the magnitude of a parameter. It's strongly related to normalized optimizers like LARS or LAMB, but operates per-output-unit and is relaxed in that it doesn't ignore gradient magnitudes.
Oct 1, 2018 • 11 tweets • 5 min read
Large-Scale GAN Training: My internship project with Jeff and Karen.

We push the SOTA Inception Score from 52 -> 166+ and give GANs the ability to trade sample variety and fidelity.

arxiv.org/abs/1809.11096 The Truncation Trick we introduce allows for direct control of sample quality (at the cost of overall variety) by pushing samples towards the mode: