Appendix A Evaluation determinism

本文为论文《s1：用简单方法实现测试时计算扩展》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

We run our evaluations using vLLM (Kwon et al., 2023) as it is faster than the alternatives we tried. However, we find that even when using the same random seeds and greedy sampling, evaluation scores can change significantly across runs:

• •

Different batch sizes causing different results see https://github.com/vllm-project/vllm/issues/5898

• •

Continuing generations causing different results see https://github.com/vllm-project/vllm/issues/11783

• •

Changes in tensor parallelism causing different results

As our model generates long reasoning traces prior to its answer, small numeric changes can snowball into large differences. We encounter many generations that are exactly the same for thousands of tokens and then suddenly differ in one token eventually ending up with an entirely different answer. To partly counter this issue we generally run our final evaluations using full precision unless otherwise indicated.