How far can we compress billion-parameter LLMs? We introduce requential coding, which achieves < 1-bit per param compression, and explains why scaling doesn't hit a generalization wall!

w/@m_finzi, @YujiaZheng9 ,@kunkzhang, @andrewgwils
1/🧵 arxiv.org/pdf/2607.11883 Existing compressors are blind to what a model actually learns. Post-training quantization pays at least the model size, while prequential coding pays for the full data entropy. Requential coding codes only the signal for improving the model, yielding much shorter codes. 2/🧵