cs phd student @harvard
deep learning theory, optimization
Jul 29 • 8 tweets • 3 min read
1/8 How well does the simplest model in optimization theory, the quadratic, actually describe LLM pretraining? Surprisingly well. We linearize a 150M model at checkpoints throughout training and the local Taylor expansion tracks the true loss for up to 10% of the training budget.
2/8 We train on 3B tokens, save checkpoints every 10%. From each, we continue training on (a) the linearized model with CE loss (NTK-style) and (b) the quadratic Taylor expansion, and compare to the real run. Late in training, the agreement is remarkable.