Appendix A Training setup

本文为论文《SmolLM2：紧凑高效的小语言模型家族》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

Table 6 shows the architecture details of SmolLM2 1.7B.

Table 6: Overview of the architecture of SmolLM2. † This is before extending the context to 8k tokens.

Figure 3 shows the progression of the learning rate through the training using WSD scheduler.

Figure 3: Learning rate during SmolLM2 training. We used WSD scheduler with 2000 steps warmup, learning rate $5.0\times 10^{-4}$ and 10% decay.