1/8 How well does the simplest model in optimization theory, the quadratic, actually describe LLM pretraining? Surprisingly well. We linearize a 150M model at checkpoints throughout training and the local Taylor expansion tracks the true loss for up to 10% of the training budget.
2/8 We train on 3B tokens, save checkpoints every 10%. From each, we continue training on (a) the linearized model with CE loss (NTK-style) and (b) the quadratic Taylor expansion, and compare to the real run. Late in training, the agreement is remarkable. 
3/8 These quadratic problems are not arbitrary and have special structure. We run Lanczos quadrature with extremely deep probes (m=1200) to resolve the Hessian spectrum over 6 orders of magnitude — far deeper into the tail than prior work.
4/8 The spectrum has a striking structure, a head of roughly vocab size eigenvalues (neural collapse), and a power-law tail that is universal across batch sizes and is unaffected by the Adam preconditioner.
5/8 We probe stability: freeze the preconditioner, run preconditioned SGD from checkpoints over a (lr, bsz) grid, check what diverges. LLM training typically occurs at a stochastic edge of stability at small B and deterministic EOS at large B, matching linear stability theory.
6/8 Takeaway: near the end of training, pretraining looks like a sequence of structured quadratics. Caveats: agreement is worse early in training and worse for the constant schedule than for cosine. We’re looking into why!
7/8 Paper link (with lots of other findings): Work done with great collaborators @pranavn1008, @depen_morwani, @KempnerInst and my fantastic advisors @CPehlevan, @ShamKakade6, @alex_damian_ (who is now taking grad students at MIT!)arxiv.org/abs/2607.21716
8/8 Tagging people: @QuanquanGu, @jainprateek_, @PNetrapalli, @aaron_defazio, @BachFrancis, @konstmish, @ddrusvyat, @KempnerInst, @_arohan_, @jxbz, @uuujingfeng
• • •
Missing some Tweet in this thread? You can try to
force a refresh
