论文信息与摘要（原文）

原文标题：SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language Model arXiv：2502.02737（v1）　方向：第三批 大模型基座：开源前沿再跃迁

以下为 arXiv 官方 HTML 全文，一字不落保留原文结构、图表与参考文献，仅供学习研究参考，版权归原作者所有。

Loubna Ben Allal

Correspondence to: loubna@hf.co

Anton Lozhkov

Elie Bakouch

Gabriel Martín Blázquez

Guilherme Penedo

Lewis Tunstall

Andrés Marafioti

Hynek Kydlíček

Agustín Piqueres Lajarín

Vaibhav Srivastav

Joshua Lochner

Caleb Fahlgren

Xuan-Son Nguyen

Clémentine Fourrier

Ben Burtenshaw

Hugo Larcher

Haojun Zhao

Cyril Zakka

Mathieu Morlon

Colin Raffel

Leandro von Werra

Correspondence to: leandro@hf.co

Thomas Wolf

Correspondence to: thomas@hf.co

Abstract

While large language models have facilitated breakthroughs in many applications of artificial intelligence, their inherent largeness makes them computationally expensive and challenging to deploy in resource-constrained settings. In this paper, we document the development of SmolLM2, a state-of-the-art “small” (1.7 billion parameter) language model (LM). To attain strong performance, we overtrain SmolLM2 on ˜11 trillion tokens of data using a multi-stage training process that mixes web text with specialized math, code, and instruction-following data. We additionally introduce new specialized datasets (FineMath, Stack-Edu, and SmolTalk) at stages where we found existing datasets to be problematically small or low-quality. To inform our design decisions, we perform both small-scale ablations as well as a manual refinement process that updates the dataset mixing rates at each stage based on the performance at the previous stage. Ultimately, we demonstrate that SmolLM2 outperforms other recent small LMs including Qwen2.5-1.5B and Llama3.2-1B. To facilitate future research on LM development as well as applications of small LMs, we release both SmolLM2 as well as all of the datasets we prepared in the course of this project.

Keywords: Machine Learning, ICML

††affiliationnotice: Equal contribution

Hugging Face

https://hf.co/collections/HuggingFaceTB/smollm2-6723884218bcda64b34d7db9