3 Pretraining datasets

本文为论文《SmolLM2：紧凑高效的小语言模型家族》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

Pretraining data curation is especially important for small LMs due to their tendency to be more sensitive to noise in the training data (Rolnick et al., 2017; Abdin et al., 2024a). In addition, designing a pretraining strategy involves not only selecting and curating data, but also determining how much to “mix” (i.e. sample) from different sources, which can be particularly important when including e.g. specialized math and code datasets. We therefore undertook a careful evaluation of existing datasets and, wherever we deemed necessary, created new, improved, and larger datasets.

3.1 Ablation setup

To compare English web datasets and find the best mixture for training our models, we followed an empirical approach similar to Penedo et al. (2024a). Specifically, we trained models on each dataset under identical conditions: model configuration, training hyperparameters, and token count. We trained 1.7B parameter Transformers (Vaswani et al., 2017) based on the Llama architecture (Touvron et al., 2023), with a sequence length of 2048, a global batch size of approximately 2 million tokens, the GPT-2 tokenizer (Radford et al., 2019), and a cosine learning rate schedule (Loshchilov & Hutter, 2016) with a learning rate of $3.0\times 10^{-4}$. Each dataset ablation model is trained on 350B tokens randomly sampled from the full dataset. For evaluation, we also followed Penedo et al. (2024a), and used lighteval to evaluate on a variety of knowledge, reasoning, and text understanding benchmarks: MMLU (Hendrycks et al., 2021), HellaSwag (Zellers et al., 2019), OpenBook QA (Mihaylov et al., 2018), PIQA (Bisk et al., 2019), WinoGrande (Sakaguchi et al., 2019), ARC (Clark et al., 2018), and CommonSenseQA (Talmor et al., 2019).

Math and code capabilities typically emerge only after extensive training, so similarly to Blakeney et al. (2024); Dubey et al. (2024); Ai2 (2024), when evaluating math and code datasets we started from a mid-training checkpoint of SmolLM2 at 3T tokens (detailed in Section 4), which was trained primarily on web data. We then used an annealing approach: the learning rate linearly decays to 0 while training on a mixture that includes the dataset under evaluation. For math, we annealed on a mixture of 60B tokens of the dataset under evaluation and 40B from the pre-checkpoint mixture. For code ablations, we performed annealing on 200B tokens, uniformly distributed across 15 of the most commonly used programming languages (˜14B tokens each). We evaluated the math ablation models on GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021) and MMLU-STEM to assess their math capabilities using lighteval, and we used HumanEval (Chen et al., 2021) and MultiPL-E (Cassano et al., 2022) to evaluate the code ablation models using the BigCode-Evaluation-Harness.

3.2 English web data

Web text from Common Crawl has remained a popular source of pretraining data, and recent classifier-based filtering techniques have significantly advanced pretraining data quality (Dubey et al., 2024; Abdin et al., 2024b; Abdin et al., 2024a; Kong et al., 2024). Two prominent examples of open datasets that use classifer-based filtering are FineWeb-Edu (Penedo et al., 2024a) and DCLM (Li et al., 2024c). FineWeb-Edu consists of 1.3T tokens that were deemed “educational” by a classifier trained on annotations generated by Llama3-70B-Instruct (Dubey et al., 2024). DCLM comprises 3.8T tokens filtered using a fastText classifier (Joulin et al., 2016a; Joulin et al., 2016b) trained on instruction-following data from OpenHermes 2.5 (Teknium, 2023a) and high-scoring posts from the r/ExplainLikeImFive (ELI5) subreddit. Training ablation models on 350B tokens each from FineWeb-Edu and DCLM attained performance shown in Table 1. We find that FineWeb-Edu achieves higher scores on the educational benchmarks MMLU, ARC, and OpenBookQA, while DCLM performs better on HellaSwag and CommonsenseQA. These results align with the datasets’ content: FineWeb-Edu prioritizes educational material, while DCLM captures more diverse, conversational styles.

Table 1: Evaluation of models trained on FineWeb-Edu and DCLM for 350B tokens. 40/60 and 60/40 denote the FW-Edu/DCLM ratio.

Given the complementary strengths of FineWeb-Edu and DCLM, we explored whether mixing them could further improve performance. After testing different ratios, we found that a 60% FineWeb-Edu and 40% DCLM mix works well, as shown in Table 1: It nearly matches FineWeb-Edu’s performance on MMLU, ARC, and OpenBookQA while also aligning with DCLM’s results on HellaSwag and approaching its performance on CommonSenseQA. Combining these datasets yields 5.1T tokens of (English) text.

3.3 Math data

Specialized math pretraining data is crucial for developing robust mathematical understanding. Recent research has shown that carefully curated mathematical content from Common Crawl, combined with targeted filtering techniques, can significantly enhance language models’ mathematical reasoning capabilities (Dubey et al., 2024; Yang et al., 2024c; Shao et al., 2024; Han et al., 2024).

3.3.1 Comparison of Existing Datasets

We compare two leading publicly available math datasets: OpenWebMath (OWM) (Paster et al., 2023) and InfiMM-WebMath (Han et al., 2024). OWM consists of 12B tokens, built by filtering math-specific content from Common Crawl and using a specialized text extraction pipeline to preserve mathematical formatting and equations. InfiMM-WebMath contains 40B text tokens, and its authors show that it matches the performance of the private dataset of DeepSeekMath (Shao et al., 2024).

We performed annealing ablations (following the setup described in Section 3.1) on OWM and InfiMM-WebMath, finding that InfiMM-WebMath achieves a peak accuracy of 14% on GSM8K compared to OWM’s 10%, while OWM slightly outperforms InfiMM-WebMath on MATH. The full evaluation curves are available in Section C.1. Despite training on 60B math tokens (i.e., 5 epochs for OWM and 1.5 epochs for InfiMM-WebMath), performance still lagged behind proprietary state-of-the-art small models (Yang et al., 2024b). Further analysis highlighted two key limitations: insufficient dataset sizes, and insufficient focus on step-by-step mathematical reasoning, along with an overrepresentation of academic papers that focus on advanced concepts.

3.3.2 New dataset: FineMath

The aforementioned issues with OWM and InfiMM-WebMath motivated us to develop FineMath11 1

https://huggingface.co/datasets/HuggingFaceTB/finemath, a collection of up to 54B tokens of math data focusing on mathematical deduction and reasoning through classifier-based filtering.

We began by extracting text from Common Crawl WARC files using Resiliparse, focusing on all 5.8B unique URLs from the FineWeb dataset (a subset of Common Crawl’s 75B unique URLs). We then employed the FineWeb-Edu filtering approach, using Llama-3.1-70B-Instruct (Dubey et al., 2024) with a prompt (Section C.2) that scores content on a 3-point scale, where 1 indicates some mathematical content and 3 indicates step-by-step problem solutions at an appropriate level. After training a classifier on these silver labels, we identified domains containing at least 10 pages with a quality score of 2 or higher. We expanded our domain coverage by including domains with at least 10 URLs from either OWM or InfiMM-WebMath. From the Common Crawl index, we retrieved a total of 7.7B URLs belonging to this list of domains: 5.7B identified by our classifier, 0.6B from OWM, and 1.3B from InfiWebMath. We then re-extracted all identified pages using the OWM pipeline, preserving LaTeX formatting and removing all-boilerplate pages, yielding 7.1B pages containing 6.5T tokens.

To retain only high-quality math content, we reapplied a classifier trained on Llama-3.1-70B-Instruct annotations using a 5-point scale prompt (Section C.3) specifically targeting pages with reasoning and middle- to high-school-level content. We note that InfiMM-WebMath used a similar classifier filtering pipeline, but their prompt did not target the same type of content. After classification, we performed deduplication using single-band MinHash LSH (Broder, 1997) with 10 hashes and applied fastText language classification (Joulin et al., 2016a; Joulin et al., 2016b) to retain only English content.

Ultimately, we developed multiple variants of FineMath, including FineMath4+ (10B tokens, 6.7M documents) which retains only samples with scores of 4-5 and FineMath3+ (34B tokens, 21.4M documents) which includes scores 3-5. We additionally applied the same classifier to InfiMM-WebMath, creating Infi-WebMath4+ (8.5B tokens, 6.3M documents) and Infi-WebMath3+ (20.5B tokens, 13.9M documents). Similarly to Yang et al. (2024c), we decontaminate each dataset against GSM8K, MATH and MMLU using 13-gram matching and a minimum overlap ratio with the longest common subsequence of 0.6.

Results

Figure 1 presents our FineMath annealing ablations. All FineMath subsets consistently outperform OWM and InfiMM-WebMath on GSM8K, MATH, and MMLU-STEM. FineMath4+ achieves a 2x improvement on GSM8K and a 6x improvement on MATH compared to InfiMM-WebMath, demonstrating the importance of retaining high-quality mathematical content with reasoning. Additionally, Infi-WebMath4+ outperforms InfiMM-WebMath, but plateaus after 80B tokens (roughly 10 epochs), likely due to data repetition, a trend not seen in FineMath4+.

Figure 1: Performance of models trained on different subsets of FineMath and other math datasets.

3.4 Code data

Code generation and understanding are becoming essential capabilities for modern LLMs, enabling diverse use cases such as code completion, debugging, and software design. While specialized code models (Lozhkov et al., 2024; Bai et al., 2023; Roziere et al., 2023) are optimized specifically for these tasks, general-purpose LLMs are increasingly deployed as coding assistants. Moreover, recent research has shown that including code data in pretraining enhances not only code-related capabilities but also improves natural language reasoning and world knowledge (Aryabumi et al., 2024). The Stack datasets are state-of-the-art open code datasets (Li et al., 2023a; Kocetkov et al., 2022), including Stack v1, ˜3TB of source code from public GitHub repositories; StarCoderData (Li et al., 2023a; Kocetkov et al., 2022; Lozhkov et al., 2024), a filtered subset of 250 billion tokens across 80 programming languages; Stack v2, with ˜32TB of data sourced from the Software Heritage code archive; and StarCoder2Data, the training corpus for StarCoder2 models (Lozhkov et al., 2024) with 900 billion tokens spanning more than 600 programming languages.

Stack-Edu

Recent work has shown that the FineWeb-Edu classifier-based filtering strategy can be effective for code data (Wei et al., 2024b; Allal et al., 2024). We therefore constructed Stack-Edu, a filtered variant of StarCoder2Data focusing on educational and well-documented code. Specifically, we selected the 15 largest programming languages from StarCoder2Data to match the capacity constraints of smaller models (Lozhkov et al., 2024) and ensure benchmark coverage for the ablations. This subset had ˜450 billion tokens. We then trained 15 language-specific classifiers using the StarEncoder model (Li et al., 2023a) on synthetic annotations generated by Llama3-70B-Instruct (Dubey et al., 2024) (prompt in Section D.1), which rated the educational quality on a scale from 0 to 5. Each classifier was trained on 500,000 samples and achieved an F1 score above 0.7 for most languages when applying a threshold of 3 for binary classification.

To evaluate Stack-Edu, we performed annealing ablations as described in Section 3.1. Filtering with a threshold of 3 improved performance across most languages while maintaining sufficient data, although Java performed better with threshold 2. Since Markdown is not included in the MultiPL-E benchmark, we could not determine a threshold for the dataset quantitatively; instead, we used threshold 3 based on qualitative analysis. The resulting Stack-Edu dataset contains ˜125B tokens across its 15 languages (see Section D.2). Table 2 shows the statistics of the top 4 programming languages in terms of size, and the impact of our educational filtering on MultiPL-E.

Table 2: Stack-Edu dataset statistics and MultiPL-E scores for the top 4 (in terms of size) programming languages. We use HumanEval for Python evaluation.