Appendix C FineMath

本文为论文《SmolLM2：紧凑高效的小语言模型家族》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

C.1 Public datasets comparison

Figure 5 Shows the performance of ablation models trained on OWM and InfiMM-WebMath on GSM8k and MATH.

图

Figure 5: Results of annealing ablations comparing OWM and the text component of InfiMM-WebMath. InfiMM-WebMath consistently outperforms OWM on GSM8K, while OWM has a slight advantage on MATH. Despite training on 60B math tokens (equivalent to 5 epochs for OWM and 1.5 epochs for InfiMM-WebMath), performance remains far below state-of-the-art LLMs, highlighting the need for a new math dataset.

C.2 Annotation Prompt (3-scale)

We used the following prompt template to generate the silver 3-scale annotations for FineMath using the Llama3 model:

Evaluate the following text extract for its potential usefulness for studying mathematics up to high school and early undergraduate levels. Use the following 3-point scoring system described below. Points are accumulated based on the satisfaction of each criterion:

  • Add 1 point if the extract contains some mathematical content, even if it’s not very useful for studying or is an academic paper that is too advanced.

  • Add another point if the extract demonstrates logical reasoning in a mathematical context, even if it lacks step-by-step explanations or is too advanced.

  • Award a third point if the extract is at an appropriate level (up to high school and early undergraduate levels) and contains clear mathematical deductions and step-by-step solutions to mathematical problems.

Question-answer formats (e.g., from educational websites or forums) are acceptable if they meet the criteria. Ignore any formatting errors or missing equations and make assumptions based on the overall content.

The text extract:

<EXTRACT>

After examining the extract:

  • Briefly justify your total score, up to 100 words.

  • Conclude with the score using the format: "Final score: <total points>".

C.3 Annotation Prompt (5-scale)

We used the following prompt template to generate the 5-scale annotations for FineMath using the Llama3 model during the second filtering stage:

Evaluate the following text extract for its potential usefulness for studying mathematics up to high school and early undergraduate levels. Use the following 5-point scoring system described below. Points are accumulated based on the satisfaction of each criterion:

  • Add 1 point if the extract contains some mathematical content, even if it’s not very useful for studying, or if it contains non-academic content such as advertisements and generated pages for converting weight and currencies.

  • Add another point if the extract touches on mathematical topics, even if it’s poorly written if it’s too complex such as an academic paper that is too advanced.

  • Award a third point if the extract demonstrates problem solving or logical reasoning in a mathematical context, even if it lacks step-by-step explanations.

  • Grant a fourth point if the extract is at an appropriate level (up to high school and early undergraduate levels) and contains clear mathematical deductions and step-by-step solutions to mathematical problems. It should be similar to a chapter from a textbook or a tutorial.

  • Give a fifth point if the extract is outstanding in its educational value for teaching and studying mathematics in middle school and high school. It should include very detailed and easy to follow explanations.

Question-answer formats (e.g., from educational websites or forums) are acceptable if they meet the criteria.

The text extract:

<EXTRACT>

After examining the extract:

  • Briefly justify your total score, up to 100 words.

  • Conclude with the score using the format: Final score: <total points>.