5 Post-training

本文为论文《SmolLM2：紧凑高效的小语言模型家族》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

After training the base SmolLM2 model, we followed current standard practice for maximizing performance and utility via post-training through instruction tuning and preference learning. For post-training, we leveraged existing datasets in addition to a new instruction tuning dataset called SmolTalk.

5.1 SmolTalk

Although the SmolLM2 base model outperformed other state-of-the-art base models in the 1-2B parameter range, the base model’s performance after fine-tuning on public datasets like MagPie-Pro (Xu et al., 2024) or OpenHermes2.5 (Teknium, 2023b) was lower than the post-trained versions of these other models. This observation motivated the development of SmolTalk22 2

https://huggingface.co/datasets/HuggingFaceTB/smoltalk, a new instruction-following dataset that carefully combines selected existing datasets with new synthetic datasets we developed, including the Magpie-Ultra conversational dataset as well as other specialized datasets that address specific capabilities like Smol-Constraint, Smol-Rewrite, and Smol-Summarization. All datasets were generated using Distilabel (Bartolomé Del Canto et al., 2024).

5.1.1 Conversational data

MagPie-Ultra is a multi-turn dataset created using the two-step prompting method from (Xu et al., 2024). Unlike MagPie, which used Llama-3-70B-Instruct without specific system prompts to generate two-turn conversations, MagPie-Ultra leverages the larger, more powerful model Llama-3.1-405B-Instruct-FP8 (Dubey et al., 2024). We also incorporate system prompts to guide generation, producing a balanced dataset of 1M samples with three-turn conversations. The resulting dataset was further filtered using smaller Llama models (Llama-3.1-8B-Instruct and Llama-Guard-3-8B) to ensure quality and safety of the generated instructions. We also leveraged ArmoRM (Wang et al., 2024b; Wang et al., 2024a) to score conversations for quality-based filtering, and gte-large-en-v1.5 (Zhang et al., 2024; Li et al., 2023c) to deduplicate semantically similar conversations.

We compare MagPie-Ultra to existing public supervised fine-tuning (SFT) datasets in Table 10 (Appendix F). The evaluation suite included the instruction-following and conversation benchmarks IFEval (Zhou et al., 2023) and MT-Bench (Zheng et al., 2023); reasoning in ARC Challenge; knowledge in MMLU-Pro as well as GSM8K and MATH for math evaluations. Our dataset outperforms MagPie-Pro on most benchmarks, and largely surpasses OpenHermes2.5 and UltraChat (Ding et al., 2023) on IFEval and MT-Bench.

5.1.2 Task-specific data

We developed additional task-specific datasets to further enhance model instruction-following with detailed constraints (Smol-Constraint), summarization (Smol-Summarization) and rewriting (Smol-Rewrite) capabilities. Smol-Constraint contains 36k instructions with detailed constraints similar to the ones found in IFEval (Zhou et al., 2023). Using the method from (Xu et al., 2024) with a targeted system prompt, we generated 550k instructions and responses for these instructions using Qwen2.5-72B-Instruct (Yang et al., 2024b). We then filtered out generated instructions that contained conflicting constraints or incorrect responses, resulting in 56.3k instruction-response pairs, which after decontaminating against IFEval (10 n-gram overlap), yielded 36k pairs. For Smol-Summarization and Smol-Rewrite, we first generated high-quality source texts that would serve as the basis for summarization and rewriting tasks. We synthesized a diverse collection of emails, tweets, LinkedIn posts, and notes using PersonaHub (Ge et al., 2024) and personas from the FinePersonas dataset (Argilla, 2024; Chan et al., 2024). This allowed us to generate diverse content by prompting Qwen2.5-72B-Instruct with specific system prompts and a persona description, obtaining texts with various writing styles, topics and perspectives. We then prompted Qwen2.5-72B-Instruct to summarize and rewrite the given texts, obtaining around 1M summaries and 600k rewritten texts. Adding the 3 Smol- datasets to MagPie-Ultra, (MagPie-Ultra$\overset{+}{}$) further improves IFEval performance as shown in  Table 10 (Appendix F).

5.1.3 Math data

To improve mathematical reasoning, we evaluated public math instruction datasets by fine-tuning on mixtures with 80% general instruction data (MagPie Ultra + Smol-Constraint, Smol-Rewrite, Smol-Summarization) and 20% math data from various sources. Results in Table 10 (Appendix F) highlight complementary dataset strengths: NuminaMath-CoT (Li et al., 2024b) demonstrated strong performance on MATH and MT-Bench, while MetaMathQA (Yu et al., 2023), which is also included in OpenHermes2.5, improved results on GSM8K. Based on these findings, we incorporated a combination of both datasets into SmolTalk.

5.1.4 Other specialized data

For code generation, we used Self-OSS-Starcoder2-Instruct (Wei et al., 2024a), containing 50k high-quality Python instruction-response pairs. To support system prompts, we included 30k randomly selected samples from SystemChats2.0 (Computations, 2024), and for function calling, we added 80k samples from APIGen-Function-Calling (Liu et al., 2024). Additionally, to maintain strong performance on long-context tasks, we incorporated an English subset of LongAlign (Bai et al., 2024) (3.7k samples with 8k–16k tokens). We also added 100k randomly selected OpenHermes2.5 samples due to its strong performance in knowledge (MMLU-Pro), Everyday-Conversations (Face, 2024), 2.2k casual multi-turn interactions, and Explore-Instruct (Wan et al., 2023) for rewriting. We found that incorporating these datasets with the specified number of samples effectively enhanced their target capabilities while preserving strong performance across other benchmarks.

5.2 Supervised fine-tuning (SFT)

Table 9 (Appendix F) shows the final composition of SmolTalk. We performed supervised fine-tuning of our base SmolLM2 on SmolTalk for 2 epochs, using a global batch size of 128, sequence length of 8192, and a learning rate of $3.0\times 10^{-4}$. The evaluation results after this SFT phase are available in Table 10 (Appendix F).

5.3 Alignment

For preference learning, we used Direct Preference Optimization (DPO) (Rafailov et al., 2024). We experimented with various public synthetic feedback datasets (Ivison et al., 2024) including UltraFeedback (Cui et al., 2024), UltraInteract (Yuan et al., 2024), Capybara (Daniele & Suphavadeeprasit, 2023), and ORCA (Lv et al., 2023). UltraFeedback proved the most consistently effective across benchmarks, improving MT-Bench, MMLU-Pro, and MATH. We trained for 2 epochs with a learning rate of $1.0\times 10^{-6}$, beta of 0.5, global batch size of 128, and sequence length of 1024 tokens. After this final stage of DPO training, we obtain the instruct SmolLM2 model. As noted in Dubey et al. (2024), using short-context data for DPO did not impact the model’s 8k context ability.

5.4 Instruct model evaluation

We evaluate the final instruct version of SmolLM2 and compare it with the instruct variants of Qwen2.5-1.5B and Llama3.2-1B, with results shown in Table 5. SmolLM2-Instruct shows strong instruction following capabilities, strongly outperforming Qwen2.5-1.5B-Instruct on IFEval; our model is competitive on MT-Bench and OpenRewrite-Eval (Shu et al., 2024) for text rewriting, and demonstrates strong mathematical capabilities as evidenced by the GSM8K and MATH scores. These results highlight SmolLM2’s ability to generalize across a variety of tasks, showcasing its potential as a capable chat assistant.

Table 5: Comparison of 1-2B instruction-tuned models across benchmarks. SmolLM2-1.7B-Instruct exhibits strong performance in instruction-following, reasoning, and math.