Alex Meterez Profile picture
cs phd student @harvard deep learning theory, optimization
Jul 29 8 tweets 3 min read
1/8 How well does the simplest model in optimization theory, the quadratic, actually describe LLM pretraining? Surprisingly well. We linearize a 150M model at checkpoints throughout training and the local Taylor expansion tracks the true loss for up to 10% of the training budget. 2/8 We train on 3B tokens, save checkpoints every 10%. From each, we continue training on (a) the linearized model with CE loss (NTK-style) and (b) the quadratic Taylor expansion, and compare to the real run. Late in training, the agreement is remarkable. Image