Appendix E Detailed pretraining results

Appendix E Detailed pretraining results
- E.1 Evaluation after each training stage
- E.2 MMLU progression

本文为论文《SmolLM2：紧凑高效的小语言模型家族》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

E.1 Evaluation after each training stage

Table 8 shows the evaluation results of SmolLM2 at the end of each training stage. In addition to the benchmarks used during the ablations, we added four generative tasks: CoQA (Reddy et al., 2019), DROP (Dua et al., 2019), Jeopardy (MosaicML, 2024) and SQuAD v2 (Rajpurkar et al., 2018)

Table 8: Per-benchmark model performance across training stages. Stages 1-3 are during stable phase (no learning rate decay).

E.2 MMLU progression

Figure 6 shows the progression of MMLU scores throughout the stable phase.

Figure 6: Progression of MMLU MCF and MLU CF during the training. We observe above-random (>25%) accuracy on MMLU MCF after 6T tokens of training, while MMLU CF appears to plateau.