SiTU-GLU

Kimi K3 Open Model Surpasses Closed AI Leaders with Million-Token Context and 896 Experts

Kimi K3 is an open-weights model that outperforms closed systems from GPT and Anthropic on several independent benchmarks. Developers achieved this by scaling three dimensions simultaneously: one million tokens of context, deeper layers, and 896 Mixture-of-Experts specialists. The team replaced the standard KV-cache with a fixed-size state matrix updated via the delta-rule, solving numerical instability by clamping the forgetting coefficient alpha between e^-5 and 1. Sequential state updates were parallelized across chips, GPUs, and multi-GPU nodes using the open-source FlashKDA kernel built on CUTLASS. Inter-layer information flow was preserved through learned pseudo-queries that selectively retrieve relevant outputs from prior blocks rather than passing full activations. Load balancing across 896 routed experts uses per-expert quantile thresholds and a distributed histogram method to guarantee fair token allocation without full sorting.