1 Introduction

本文为论文《SmolLM2：紧凑高效的小语言模型家族》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

Large language models (LMs) have become a cornerstone of modern AI systems due to their ability to follow natural language instructions and flexibly perform a huge range of tasks (Touvron et al., 2023; Bai et al., 2023; Brown et al., 2020; Dubey et al., 2024; Groeneveld et al., 2024; Chowdhery et al., 2023; Young et al., 2024; Taylor et al., 2022). LLMs are, by their nature, large, in the sense that they are models with many parameters (more than ˜10 billion, by current conventions). This enormity results in enormous computational costs, both during training and for inference, which can prevent LLMs from being used in resource-constrained settings. To address this issue, a flurry of recent work has aimed to produce performant small (˜3 billion parameters or less) LMs (Gunter et al., 2024; Yang et al., 2024b; AI@Meta, 2024b; Team et al., 2024; Li et al., 2023b). These small LMs are computationally inexpensive and can be run on a wider range of devices (e.g. mobile phones) while providing satisfactory performance on many important tasks.

A key factor in the performance and behavior of LMs is the data used to train them. While important for an LM of any size, data curation has an especially outsized influence for smaller models, as their limited capacity must be carefully optimized for learning core knowledge and fundamental capabilities rather than memorizing incidental facts (Abdin et al., 2024a; Rolnick et al., 2017). Most LMs are primarily trained on text crawled from the web (Radford et al., 2019; Raffel et al., 2020) and state-of-the-art pipelines include sophisticated filtering and processing stages that aim to improve data quality (Li et al., 2024c; Penedo et al., 2024b; Penedo et al., 2024a; Soldaini et al., 2024). Recently, it has become common to include “specialized” data from certain domains such as software code (Kocetkov et al., 2022; Lozhkov et al., 2024) and mathematics (Paster et al., 2023; Han et al., 2024), which can improve performance not only on those domains but also more generally on challenging tasks that require reasoning (Muennighoff et al., 2023; Aryabumi et al., 2024).

Motivated by the above considerations, our contributions in this paper are as follows: First, we perform a careful evaluation of existing web, code, math, and instruction-following datasets (Section 3) to help guide training data design choices, ultimately training SmolLM2 via a multi-stage manual rebalancing of different sources to maximize performance (Section 4). Such on-the-fly rebalancing is a promising approach for large-scale training runs which can be sufficiently costly (around 1e23 FLOPs, or $250,000 USD worth of GPU compute for SmolLM2) to preclude running multiple full-scale training runs. Following standard practice, we also develop an instruction-tuned variant of SmolLM2 (Section 5). Additionally, after finding that existing datasets were too small and/or low-quality, we created the new datasets FineMath, Stack-Edu, and SmolTalk (for mathematics, code, and instruction-following respectively). Ultimately, we show that both the base and instruction-tuned variants of SmolLM2 are state-of-the-art among similarly sized models (Section 4.7 and Section 5.4).