π "Quantization is not a compromise β it's the next paradigm."
After K2-Thinking's release, many developers have been curious about its native INT4 quantization format.
εε°δΌ, infra engineer at @Kimi_Moonshot and Zhihu contributor, shares an insider's view on why this choice matters β and why quantization today isn't just about sacrificing precision for speed.
π‘ Key idea
In the context of LLMs, quantization is no longer a trade-off.
With the evolution of param-scaling and test-time-scaling, native low-bit quantization will become a standard paradigm for large model training.
π€ Why Low-bit Quantization Matters
In modern LLM inference, there are two distinct optimization goals:
β’ High throughput (cost-oriented): maximize GPU utilization via large batch sizes.
β’ Low latency (user-oriented): minimize per-query response time.
For Kimi-K2's MoE structure (with 1/48 sparsity), decoding is memory-bound β the smaller the model weights, the faster the compute.
FP8 weights (β1 TB) already hit the limit of what a single high-speed interconnect GPU node can handle.
β οΈ By switching to W4A16, latency drops sharply while maintaining quality β a perfect fit for low-latency inference.
π Why QAT over PTQ
Post-training quantization (PTQ) worked well for shorter generations, but failed in longer reasoning chains:
β’ Error accumulation during long decoding degraded precision.
β’ Dependence on calibration data caused "expert distortion" in sparse MoE layers.
βΌοΈThus, K2-Thinking adopted QAT for minimal loss and more stable long-context reasoning.
π§ How it works
K2-Thinking uses a weight-only QAT with fake quantization + STE (straight-through estimator).
The pipeline was fully integrated in just days β from QAT training β INT4 inference β RL rollout β enabling near lossless results without extra tokens or retraining.
β‘ INT4's hidden advantage in RL
Few people mention this: native INT4 doesn't just speed up inference β it accelerates RL training itself.
Because RL rollouts often suffer from "long-tail" inefficiency, INT4's low-latency profile makes those stages much faster.
In practice, each RL iteration runs 10-20% faster end-to-end.
Moreover, quantized RL brings stability: smaller representational space reduces accumulation error, improving learning robustness.
π© Why INT4, not MXFP4
Kimi chose INT4 over "fancier" MXFP4/NVFP4 to better support non-Blackwell GPUs, with strong existing kernel support (e.g., Marlin).
At a quant scale of 1Γ32, INT4 matches FP4 formats in expressiveness while being more hardware-adaptable.
π§ Looking forward
W4A16 is just the beginning β W4A8 and even W4A4 are on the horizon.
As new chips roll out with FP4-native operators, Kimi's quantization path will continue evolving.
"In the LLM age, quantization stands alongside SOTA and Frontier.
It's not a patch β it's how we'll reach the frontier faster."
π Full article (in Chinese): zhihu.com/question/19695β¦
#KimiK2Thinking #INT4 #Quantization #LLM #Infra #RLHF
After K2-Thinking's release, many developers have been curious about its native INT4 quantization format.
εε°δΌ, infra engineer at @Kimi_Moonshot and Zhihu contributor, shares an insider's view on why this choice matters β and why quantization today isn't just about sacrificing precision for speed.
π‘ Key idea
In the context of LLMs, quantization is no longer a trade-off.
With the evolution of param-scaling and test-time-scaling, native low-bit quantization will become a standard paradigm for large model training.
π€ Why Low-bit Quantization Matters
In modern LLM inference, there are two distinct optimization goals:
β’ High throughput (cost-oriented): maximize GPU utilization via large batch sizes.
β’ Low latency (user-oriented): minimize per-query response time.
For Kimi-K2's MoE structure (with 1/48 sparsity), decoding is memory-bound β the smaller the model weights, the faster the compute.
FP8 weights (β1 TB) already hit the limit of what a single high-speed interconnect GPU node can handle.
β οΈ By switching to W4A16, latency drops sharply while maintaining quality β a perfect fit for low-latency inference.
π Why QAT over PTQ
Post-training quantization (PTQ) worked well for shorter generations, but failed in longer reasoning chains:
β’ Error accumulation during long decoding degraded precision.
β’ Dependence on calibration data caused "expert distortion" in sparse MoE layers.
βΌοΈThus, K2-Thinking adopted QAT for minimal loss and more stable long-context reasoning.
π§ How it works
K2-Thinking uses a weight-only QAT with fake quantization + STE (straight-through estimator).
The pipeline was fully integrated in just days β from QAT training β INT4 inference β RL rollout β enabling near lossless results without extra tokens or retraining.
β‘ INT4's hidden advantage in RL
Few people mention this: native INT4 doesn't just speed up inference β it accelerates RL training itself.
Because RL rollouts often suffer from "long-tail" inefficiency, INT4's low-latency profile makes those stages much faster.
In practice, each RL iteration runs 10-20% faster end-to-end.
Moreover, quantized RL brings stability: smaller representational space reduces accumulation error, improving learning robustness.
π© Why INT4, not MXFP4
Kimi chose INT4 over "fancier" MXFP4/NVFP4 to better support non-Blackwell GPUs, with strong existing kernel support (e.g., Marlin).
At a quant scale of 1Γ32, INT4 matches FP4 formats in expressiveness while being more hardware-adaptable.
π§ Looking forward
W4A16 is just the beginning β W4A8 and even W4A4 are on the horizon.
As new chips roll out with FP4-native operators, Kimi's quantization path will continue evolving.
"In the LLM age, quantization stands alongside SOTA and Frontier.
It's not a patch β it's how we'll reach the frontier faster."
π Full article (in Chinese): zhihu.com/question/19695β¦
#KimiK2Thinking #INT4 #Quantization #LLM #Infra #RLHF
β’ β’ β’
Missing some Tweet in this thread? You can try to
force a refresh
γ
