How far can we compress billion-parameter LLMs? We introduce requential coding, which achieves < 1-bit per param compression, and explains why scaling doesn't hit a generalization wall!
w/@m_finzi, @YujiaZheng9 ,@kunkzhang, @andrewgwils
1/🧵 arxiv.org/pdf/2607.11883
Existing compressors are blind to what a model actually learns. Post-training quantization pays at least the model size, while prequential coding pays for the full data entropy. Requential coding codes only the signal for improving the model, yielding much shorter codes. 2/🧵
We code a student P that generates its own training data, with a teacher Q selecting which samples to actually train P on. Using relative entropy coding, student’s code records only these selections w/ ~KL(Q||P) bits per sample, and the selected data is distributed as Q.
3/🧵
Such a powerful compressor reveals a range of phenomena inaccessible to previous methods. We show that larger models and ensembles can be compressed to smaller sizes at fixed loss despite having more parameters, breaking 1-bit per param floor achievable by standard PTQ. 4/🧵
Compression implies generalization. Plugged into a PAC-Bayes bound, our code shows LLMs provably generalize better with scale both when holding dataset fixed and trained compute optimally, where code bits per param and certified generalization gap vanish as a power law!
5/🧵
It’s worth pausing to reflect on the significance of this finding. If this were not true, the modern scaling paradigm could have run into a wall set by an irreducible generalization gap where further scaling lowers training loss but no longer improves test performance. 6/🧵
The requential code also sheds light on a range of other key generalization phenomena. It predicts the gradual buildup of overfitting under multi-epoch training due to memorization, a process invisible to parameter-based compression. 7/🧵
Its code length measures how much information in a dataset is learnable to the model, i.e., epiplexity, rather than random, unpredictable content, a question the prequential code fails to answer and is inaccessible to parameter-based compression. Much more in the paper! 8/🧵
Much more in the paper! Check out our code at
9/🧵github.com/shikaiqiu/requ…
Share this Scrolly Tale with your friends.
A Scrolly Tale is a new way to read Twitter threads with a more visually immersive experience.
Discover more beautiful Scrolly Tales like this.