Appendix D QK-Clip Does Not Impair Model Quality

本文为论文《Kimi K2：面向智能体能力的开源大模型》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

The QK-Clip design follows a minimal intervention principle: it activates only when necessary, and deactivates after training stabilizes. Empirical evidence and analysis converge on its negligible impact on model quality.

Small-Scale Ablations

Figure 12: Applying QK-Clip to Muon in a small-scale setting with an aggresive threshold ($\tau$ = 30) has negligible impact on loss, indicating that it is a safe and effective method for constraining attention logits.

We train two small-scale 0.5B activated and 3B total parameters MoE models, one with vanilla Muon and the other with MuonClip using a low clipping threshold ($\tau=30$). As shown in Figure 12, applying MuonClip has negligible effects on the loss curve, indicating that even aggressive clipping does not impair convergence or training dynamics with MuonClip. This demonstrates that MuonClip is a safe and effective method for bounding attention logits without degrading model performance. Furthermore, evaluation on downstream tasks reveals no statistically significant degradation in performance. These results collectively demonstrate that MuonClip is a safe and effective method for bounding attention logits without compromising model quality.

Self-deactivation

In Kimi K2, QK-Clip was only transiently active:

• •

Initial 70000 steps: $12.7\%$ of attention heads triggered QK-Clip for at least once, clamping $S_{\max}$ to $100$.

• •

Post-70000 steps: All heads at some point reduced their $S_{\max}$ below $100$, rendering QK-Clip inactive.

When QK-Clip is active, it is applied per-head (rather than per-layer) to minimize potential over-regularization on other heads. After training stabilizes, QK-clip is deactivated and has no effect at all.