Kimi Linear Tech Report is dropped! ๐
huggingface.co/moonshotai/Kimโฆ
Kimi Linear: A novel architecture that outperforms full attention with faster speeds and better performanceโready to serve as a drop-in replacement for full attention, featuring our open-sourced KDA kernels! Kimi Linear offers up to a 75% reduction in KV cache usage and up to 6x decoding throughput at a 1M context length.
Key highlights:
๐น Kimi Delta Attention: A hardware-efficient linear attention mechanism that refines the gated delta rule.
๐น Kimi Linear Architecture: The first hybrid linear architecture to surpass pure full attention quality across the board.
๐น Empirical Validation: Scaled, fair comparisons + open-sourced KDA kernels, vLLM integration, and checkpoints.
The future of agentic-oriented attention is here! ๐ก
huggingface.co/moonshotai/Kimโฆ
Kimi Linear: A novel architecture that outperforms full attention with faster speeds and better performanceโready to serve as a drop-in replacement for full attention, featuring our open-sourced KDA kernels! Kimi Linear offers up to a 75% reduction in KV cache usage and up to 6x decoding throughput at a 1M context length.
Key highlights:
๐น Kimi Delta Attention: A hardware-efficient linear attention mechanism that refines the gated delta rule.
๐น Kimi Linear Architecture: The first hybrid linear architecture to surpass pure full attention quality across the board.
๐น Empirical Validation: Scaled, fair comparisons + open-sourced KDA kernels, vLLM integration, and checkpoints.
The future of agentic-oriented attention is here! ๐ก
โข โข โข
Missing some Tweet in this thread? You can try to
force a refresh
ใ
