Similar to DeepSeek in January 2025, Panicans may think that the AI networking switch TAM will massively shrink because Kimi K3 uses KDA Attention, which reduces KV-transfer networking bandwidth by up to 10x. But the opposite is true, as we explain below. 👇️ 1/8🧵
While it is true that Kimi K3 uses Kimi Delta Linear Attention (KDA) in 3 out of every 4 layers and that KDA reduces KV-cache transfer bandwidth by up to 10x compared with comparable full global-attention models, the important missing piece is that Kimi K3 requires WideEP to serve. 2/8🧵
Because Kimi K3 has 2.8 trillion parameters, even at MXFP4, each forward pass will require 1.5 TB of HBM bandwidth. This means that, even with spec decode, serving it profitably at a reasonable level of interactivity requires aggregating many chips together over a high-bandwidth network, such as the GB300 NVL72. 3/8🧵
WideEP is a very network-intensive serving optimization that Kimi K3 will need to use. WideEP works by spreading 896 experts across many GPUs within a scale-up domain, then dispatching tokens to the correct GPU and combining the results. 4/8🧵
WideEP dispatch and combine need to happen twice on every layer and for every forward pass. That means that, for each forward pass, dispatch and combine will need to run 120+ times. 5/8🧵
KV-cache transfer between prefill and decode happens only once per turn, while WideEP happens 120+ times per output forward pass, with potentially 500+ output tokens per turn. Thus, the relative KV-cache transfer savings from KDA linear attention are dwarfed by the increase in scale-up bandwidth required for Kimi K3’s massive count of 896 experts. 6/8🧵
Furthermore, with optimizations like incremental KV-cache transfers, prefill only needs to transfer the portions cached by the non-decode instance, so KV transfer does not take up much networking bandwidth relative to WideEP, even before KDA. 7/8🧵
More efficient attention will further push context lengths from 1M to 5M+, with less context rot. Jevons’ Paradox means that making attention more efficient will lead to wider AI adoption, which will require more networking. 8/8🧵
Share this Scrolly Tale with your friends.
A Scrolly Tale is a new way to read Twitter threads with a more visually immersive experience.
Discover more beautiful Scrolly Tales like this.