6 SmolLM2 135M and 360M

本文为论文《SmolLM2：紧凑高效的小语言模型家族》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

In addition to SmolLM2-1.7B, we also trained two smaller models: SmolLM2-360M (360M parameters, trained on 4T tokens) and SmolLM2-135M (135M parameters, trained on 2T tokens), which are similarly state-of-the-art in their size class. Given their smaller capacity and reduced training cost, we re-ran data ablations at the target training length to determine the most effective data mixture. We found that filtering DCLM with the FineWeb-Edu classifier, removing samples with score 0, and downsampling those with scores 1 and 2 worked best. Unlike SmolLM2-1.7B, where we leveraged a multi-stage training strategy, these smaller models benefited from a single-stage training approach with consistently high-quality data. We incorporated Stack-Edu from the start, alongside InfiMM-WebMath, FineMath, and Cosmopedia. These models share the same architecture as SmolLM2-1.7B but use Grouped Query Attention (GQA) and were trained using the WSD scheduler with 20% decay and a learning rate of $3.0\times 10^{-3}$. For post-training, we applied SFT using a filtered version of SmolTalk33 3

https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk, removing complex instruction-following tasks (e.g., function calling) and hard examples from MagPie-Ultra to better align with the models’ capacity. Finally, we performed DPO training using UltraFeedback, optimizing the models for instruction-following while preserving coherence and helpfulness. More details about SmolLM2-360M and 135M can be found in their respective model cards44 4

SmolLM2-360M model card55 5

SmolLM2-135M model card.