Introducing π¨ππππππππ πΉππππ
ππππ: Rethinking depth-wise aggregation.
Residual connections have long relied on fixed, uniform accumulation. Inspired by the duality of time and depth, we introduce Attention Residuals, replacing standard depth-wise recurrence with learned, input-dependent attention over preceding layers.
πΉ Enables networks to selectively retrieve past representations, naturally mitigating dilution and hidden-state growth.
πΉ Introduces Block AttnRes, partitioning layers into compressed blocks to make cross-layer attention practical at scale.
πΉ Serves as an efficient drop-in replacement, demonstrating a 1.25x compute advantage with negligible (<2%) inference latency overhead.
πΉ Validated on the Kimi Linear architecture (48B total, 3B activated parameters), delivering consistent downstream performance gains.
πFull report:
github.com/MoonshotAI/Attβ¦
Residual connections have long relied on fixed, uniform accumulation. Inspired by the duality of time and depth, we introduce Attention Residuals, replacing standard depth-wise recurrence with learned, input-dependent attention over preceding layers.
πΉ Enables networks to selectively retrieve past representations, naturally mitigating dilution and hidden-state growth.
πΉ Introduces Block AttnRes, partitioning layers into compressed blocks to make cross-layer attention practical at scale.
πΉ Serves as an efficient drop-in replacement, demonstrating a 1.25x compute advantage with negligible (<2%) inference latency overhead.
πΉ Validated on the Kimi Linear architecture (48B total, 3B activated parameters), delivering consistent downstream performance gains.
πFull report:
github.com/MoonshotAI/Attβ¦
Scaling law experiments reveal a consistent 1.25Γ compute advantage across varying model sizes.
Analysis of training dynamics demonstrates how AttnRes naturally mitigates hidden-state magnitude growth and yields a more uniform gradient distribution across depth.
For more details, check out our paper here: github.com/MoonshotAI/Attβ¦
β’ β’ β’
Missing some Tweet in this thread? You can try to
force a refresh
γ
