2 Background

本文为论文《SmolLM2：紧凑高效的小语言模型家族》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

Training a modern LM typically begins with “pretraining” on a large amount (e.g. trillions of tokens) of unstructured text. Pretraining helps the model fit the structure of language (Clark, 2019) and store factual knowledge (Petroni et al., 2019; Roberts et al., 2020) and therefore has proven to be a vital part of LM training, which made the composition of the pretraining dataset a key consideration. The data-hungry nature of pretraining has led to the use of large-scale web scrapes (com, ; ope, ; ant, ) which in their raw form can lead to poorly performing LMs (Penedo et al., 2024b). Consequently, the primary means of curation for modern LM pretraining datasets involves designing sophisticated pipelines for automatically filtering and reformatting web texts (Penedo et al., 2024a; Penedo et al., 2024b; Soldaini et al., 2024; Soboleva et al., 2023; Li et al., 2024c) that aim to keep enough data to avoid detrimental repetition (Muennighoff et al., 2023) while discarding any data that is not “high-quality”.

Apart from web text, including “specialized” data from certain domains – code (Kocetkov et al., 2022; Li et al., 2023a) and math (Paster et al., 2023; Han et al., 2024; Wang et al., ; Azerbayev et al., 2023) in particular – can improve model performance on tasks that involve reasoning and world knowledge (Muennighoff et al., 2023; Aryabumi et al., 2024; Lewkowycz et al., 2022; Shao et al., 2024). The contribution of small specialized datasets can be dwarfed by much larger web-based pretraining data sources, which has led to the design of multi-stage pretraining where specialized or high-quality datasets are incorporated later in training (Abdin et al., 2024b; Ai2, 2024; Blakeney et al., 2024; Singer et al., 2024).

After pretraining, language models typically undergo two additional training stages before deployment: instruction tuning and preference learning. In instruction tuning, the model undergoes supervised training on instruction/response pairs that reflect the way that the language model should answer a query (Wei et al., 2021; Mishra et al., 2021; Sanh et al., 2021; Wang et al., 2022). This process provides a valuable way of tailoring LMs to provide helpful responses rather than simply attempting to continue the input (as taught during pretraining). During preference learning, language models are further “aligned” towards their intended use by being trained to distinguish between helpful and unhelpful responses (Ouyang et al., 2022; Bai et al., 2022). This final stage typically involves a form of reinforcement learning (Bai et al., 2022; Lee et al., ; Rafailov et al., 2024) on data labeled with human or synthetically generated preferences.