References

本文为论文《SmolLM2：紧凑高效的小语言模型家族》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

  • (1)

Claudebot documentation.

https://darkvisitors.com/agents/claudebot.

Accessed: 2024-06-05.

  • (2)

Common crawl.

https://commoncrawl.org/.

Accessed: 2024-06-05.

  • (3)

Openai gptbot documentation.

https://platform.openai.com/docs/gptbot.

Accessed: 2024-06-05.

  • Abdin et al. (2024a)

Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al.

Phi-3 technical report: A highly capable language model locally on your phone.

arXiv preprint arXiv:2404.14219, 2024a.

  • Abdin et al. (2024b)

Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al.

Phi-4 technical report.

arXiv preprint arXiv:2412.08905, 2024b.

  • Achiam et al. (2023)

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.

Gpt-4 technical report.

arXiv preprint arXiv:2303.08774, 2023.

  • Ai2 (2024)

Ai2.

Olmo 2: The best fully open language model to date.

https://allenai.org/blog/olmo2, 2024.

Blog post.

  • AI@Meta (2024a)

AI@Meta.

Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, 2024a.

URL https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/.

  • AI@Meta (2024b)

AI@Meta.

Llama 3.2 model card, 2024b.

URL https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md.

  • Allal et al. (2024)

Allal, L. B., Lozhkov, A., Bakouch, E., von Werra, L., and Wolf, T.

Smollm - blazingly fast and remarkably powerful, 2024.

  • Almazrouei et al. (2023)

Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Goffinet, É., Hesslow, D., Launay, J., Malartic, Q., et al.

The falcon series of open language models.

arXiv preprint arXiv:2311.16867, 2023.

  • Argilla (2024)

Argilla.

Finepersonas-v0.1 dataset.

https://huggingface.co/datasets/argilla/FinePersonas-v0.1, 2024.

Available on Hugging Face Datasets.

  • Aryabumi et al. (2024)

Aryabumi, V., Su, Y., Ma, R., Morisot, A., Zhang, I., Locatelli, A., Fadaee, M., Üstün, A., and Hooker, S.

To code, or not to code? exploring impact of code in pre-training.

arXiv preprint arXiv:2408.10914, 2024.

  • Azerbayev et al. (2023)

Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S.

Llemma: An open language model for mathematics.

arXiv preprint arXiv:2310.10631, 2023.

  • Bai et al. (2023)

Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.

Qwen technical report.

arXiv preprint arXiv:2309.16609, 2023.

  • Bai et al. (2022)

Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al.

Constitutional ai: Harmlessness from ai feedback.

arXiv preprint arXiv:2212.08073, 2022.

  • Bai et al. (2024)

Bai, Y., Lv, X., Zhang, J., He, Y., Qi, J., Hou, L., Tang, J., Dong, Y., and Li, J.

Longalign: A recipe for long context alignment of large language models.

arXiv preprint arXiv:2401.18058, 2024.

  • Bartolomé Del Canto et al. (2024)

Bartolomé Del Canto, Á., Martín Blázquez, G., Piqueres Lajarín, A., and Vila Suero, D.

Distilabel: An AI feedback (AIF) framework for building datasets with and for LLMs.

https://github.com/argilla-io/distilabel, 2024.

  • Bisk et al. (2019)

Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y.

Piqa: Reasoning about physical commonsense in natural language, 2019.

  • Blakeney et al. (2024)

Blakeney, C., Paul, M., Larsen, B. W., Owen, S., and Frankle, J.

Does your data spark joy? performance gains from domain upsampling at the end of training.

arXiv preprint arXiv:2406.03476, 2024.

  • Broder (1997)

Broder, A. Z.

On the resemblance and containment of documents.

Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171), pp. 21–29, 1997.

  • Brown et al. (2020)

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.

Language models are few-shot learners.

arXiv preprint arXiv:2005.14165, 2020.

  • Cassano et al. (2022)

Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al.

Multipl-e: A scalable and extensible approach to benchmarking neural code generation.

arXiv preprint arXiv:2208.08227, 2022.

  • Chan et al. (2024)

Chan, X., Wang, X., Yu, D., Mi, H., and Yu, D.

Scaling synthetic data creation with 1,000,000,000 personas, 2024.

URL https://arxiv.org/abs/2406.20094.

  • Chen et al. (2021)

Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al.

Evaluating large language models trained on code.

arXiv preprint arXiv:2107.03374, 2021.

  • Chowdhery et al. (2023)

Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al.

Palm: Scaling language modeling with pathways.

Journal of Machine Learning Research, 24(240):1–113, 2023.

  • Clark (2019)

Clark, K.

What does bert look at? an analysis of bert’s attention.

arXiv preprint arXiv:1906.04341, 2019.

  • Clark et al. (2018)

Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O.

Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.

  • Cobbe et al. (2021)

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.

Training verifiers to solve math word problems.

arXiv preprint arXiv:2110.14168, 2021.

  • Computations (2024)

Computations, C.

Systemchat-2.0.

https://huggingface.co/datasets/cognitivecomputations/SystemChat-2.0, 2024.

  • Cui et al. (2024)

Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., et al.

Ultrafeedback: Boosting language models with scaled ai feedback.

In Forty-first International Conference on Machine Learning, 2024.

  • Daniele & Suphavadeeprasit (2023)

Daniele, L. and Suphavadeeprasit.

Amplify-instruct: Synthetically generated diverse multi-turn conversations for efficient llm training.

arXiv preprint arXiv:(coming soon), 2023.

URL https://huggingface.co/datasets/LDJnr/Capybara.

  • de Vries (2023)

de Vries, H.

Go smol or go home.

https://www.harmdevries.com/post/model-size-vs-compute-overhead/, 2023.

  • Ding et al. (2023)

Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B.

Enhancing chat language models by scaling high-quality instructional conversations.

arXiv preprint arXiv:2305.14233, 2023.

  • Du et al. (2024)

Du, Z., Zeng, A., Dong, Y., and Tang, J.

Understanding emergent abilities of language models from the loss perspective.

arXiv preprint arXiv:2403.15796, 2024.

  • Dua et al. (2019)

Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M.

Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs.

arXiv preprint arXiv:1903.00161, 2019.

  • Dubey et al. (2024)

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.

The llama 3 herd of models.

arXiv preprint arXiv:2407.21783, 2024.

  • Face (2024)

Face, H.

Everyday conversations for llms.

https://huggingface.co/datasets/HuggingFaceTB/everyday-conversations-llama3.1-2k, 2024.

  • Gao et al. (2024)

Gao, T., Wettig, A., Yen, H., and Chen, D.

How to train long-context language models (effectively), 2024.

URL https://arxiv.org/abs/2410.02660.

  • Ge et al. (2024)

Ge, T., Chan, X., Wang, X., Yu, D., Mi, H., and Yu, D.

Scaling synthetic data creation with 1,000,000,000 personas.

arXiv preprint arXiv:2406.20094, 2024.

  • Groeneveld et al. (2024)

Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al.

Olmo: Accelerating the science of language models.

arXiv preprint arXiv:2402.00838, 2024.

  • Gu et al. (2024)

Gu, Y., Tafjord, O., Kuehl, B., Haddad, D., Dodge, J., and Hajishirzi, H.

Olmes: A standard for language model evaluations.

arXiv preprint arXiv:2406.08446, 2024.

  • Gunter et al. (2024)

Gunter, T., Wang, Z., Wang, C., Pang, R., Narayanan, A., Zhang, A., Zhang, B., Chen, C., Chiu, C.-C., Qiu, D., et al.

Apple intelligence foundation language models.

arXiv preprint arXiv:2407.21075, 2024.

  • Hägele et al. (2024)

Hägele, A., Bakouch, E., Kosson, A., Allal, L. B., Von Werra, L., and Jaggi, M.

Scaling laws and compute-optimal training beyond fixed training durations.

arXiv preprint arXiv:2405.18392, 2024.

  • Han et al. (2024)

Han, X., Jian, Y., Hu, X., Liu, H., Wang, Y., Fan, Q., Ai, Y., Huang, H., He, R., Yang, Z., et al.

Infimm-webmath-40b: Advancing multimodal pre-training for enhanced mathematical reasoning.

arXiv preprint arXiv:2409.12568, 2024.

  • Hendrycks et al. (2021)

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J.

Measuring massive multitask language understanding, 2021.

  • Hoffmann et al. (2022)

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al.

Training compute-optimal large language models.

arXiv preprint arXiv:2203.15556, 2022.

  • Hu et al. (2024)

Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., Zhang, X., Thai, Z. L., Zhang, K., Wang, C., Yao, Y., Zhao, C., Zhou, J., Cai, J., Zhai, Z., Ding, N., Jia, C., Zeng, G., Li, D., Liu, Z., and Sun, M.

Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024.

URL https://arxiv.org/abs/2404.06395.

  • Ivison et al. (2024)

Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., and Hajishirzi, H.

Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback.

arXiv preprint arXiv:2406.09279, 2024.

  • Joshi et al. (2017)

Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L.

Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.

arXiv preprint arXiv:1705.03551, 2017.

  • Joulin et al. (2016a)

Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., and Mikolov, T.

Fasttext.zip: Compressing text classification models.

arXiv preprint arXiv:1612.03651, 2016a.

  • Joulin et al. (2016b)

Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T.

Bag of tricks for efficient text classification.

arXiv preprint arXiv:1607.01759, 2016b.

  • Kamradt (2024)

Kamradt, G.

Needle in a haystack - pressure testing llms.

https://github.com/gkamradt/LLMTestNeedleInAHaystack, 2024.

  • Kocetkov et al. (2022)

Kocetkov, D., Li, R., Allal, L. B., Li, J., Mou, C., Ferrandis, C. M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., et al.

The stack: 3 tb of permissively licensed source code.

arXiv preprint arXiv:2211.15533, 2022.

  • Kong et al. (2024)

Kong, X., Gunter, T., and Pang, R.

Large language model-guided document selection.

arXiv preprint arXiv:2406.04638, 2024.

  • Kwiatkowski et al. (2019)

Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al.

Natural questions: a benchmark for question answering research.

Transactions of the Association for Computational Linguistics, 7, 2019.

  • (57)

Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K. R., Bishop, C., Hall, E., Carbune, V., Rastogi, A., et al.

Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback.

In Forty-first International Conference on Machine Learning.

  • Lewkowycz et al. (2022)

Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al.

Solving quantitative reasoning problems with language models.

Advances in Neural Information Processing Systems, 35:3843–3857, 2022.

  • Li et al. (2024a)

Li, C., Yuan, Z., Yuan, H., Dong, G., Lu, K., Wu, J., Tan, C., Wang, X., and Zhou, C.

Mugglemath: Assessing the impact of query and response augmentation on math reasoning.

In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10230–10258, 2024a.

  • Li et al. (2024b)

Li, J., Beeching, E., Tunstall, L., Lipkin, B., Soletskyi, R., Huang, S. C., Rasul, K., Yu, L., Jiang, A., Shen, Z., Qin, Z., Dong, B., Zhou, L., Fleureau, Y., Lample, G., and Polu, S.

Numinamath.

https://huggingface.co/AI-MO/NuminaMath-CoT, 2024b.

  • Li et al. (2024c)

Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha, E., Keh, S., Arora, K., et al.

Datacomp-LM: In search of the next generation of training sets for language models.

arXiv preprint arXiv:2406.11794, 2024c.

  • Li et al. (2023a)

Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al.

Starcoder: may the source be with you!

arXiv preprint arXiv:2305.06161, 2023a.

  • Li et al. (2023b)

Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T.

Textbooks are all you need ii: phi-1.5 technical report.

arXiv preprint arXiv:2309.05463, 2023b.

  • Li et al. (2023c)

Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., and Zhang, M.

Towards general text embeddings with multi-stage contrastive learning.

arXiv preprint arXiv:2308.03281, 2023c.

  • Liu et al. (2024)

Liu, Z., Hoang, T., Zhang, J., Zhu, M., Lan, T., Kokane, S., Tan, J., Yao, W., Liu, Z., Feng, Y., et al.

Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets.

arXiv preprint arXiv:2406.18518, 2024.

  • Loshchilov & Hutter (2016)

Loshchilov, I. and Hutter, F.

SGDR: Stochastic gradient descent with warm restarts.

arXiv preprint arXiv:1608.03983, 2016.

  • Lozhkov et al. (2024)

Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al.

Starcoder 2 and the stack v2: The next generation.

arXiv preprint arXiv:2402.19173, 2024.

  • Lv et al. (2023)

Lv, K., Zhang, W., and Shen, H.

Supervised fine-tuning and direct preference optimization on intel gaudi2.

https://medium.com/intel-analytics-software/a1197d8a3cd3, 2023.

Intel Corporation.

  • Mihaylov et al. (2018)

Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A.

Can a suit of armor conduct electricity? a new dataset for open book question answering.

In EMNLP, 2018.

  • Mishra et al. (2021)

Mishra, S., Khashabi, D., Baral, C., and Hajishirzi, H.

Cross-task generalization via natural language crowdsourcing instructions.

arXiv preprint arXiv:2104.08773, 2021.

  • MosaicML (2024)

MosaicML.

Llm foundry - jeopardy dataset, 2024.

URL https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/local_data/world_knowledge/jeopardy_all.jsonl.

Accessed: 2024-11-10.

  • Muennighoff et al. (2023)

Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., and Raffel, C.

Scaling data-constrained language models, 2023.

  • Ouyang et al. (2022)

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.

Training language models to follow instructions with human feedback.

Advances in neural information processing systems, 35:27730–27744, 2022.

  • Paster et al. (2023)

Paster, K., Santos, M. D., Azerbayev, Z., and Ba, J.

Openwebmath: An open dataset of high-quality mathematical web text.

arXiv preprint arXiv:2310.06786, 2023.

  • Penedo et al. (2024a)

Penedo, G., Kydlícek, H., Allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., von Werra, L., and Wolf, T.

The fineweb datasets: Decanting the web for the finest text data at scale.

ArXiv, abs/2406.17557, 2024a.

URL https://api.semanticscholar.org/CorpusID:270711474.

  • Penedo et al. (2024b)

Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Alobeidli, H., Cappelli, A., Pannier, B., Almazrouei, E., and Launay, J.

The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data only.

In Advances in Neural Information Processing Systems, 2024b.

  • Petroni et al. (2019)

Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., and Riedel, S.

Language models as knowledge bases?

arXiv preprint arXiv:1909.01066, 2019.

  • Radford et al. (2019)

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I.

Language models are unsupervised multitask learners.

2019.

  • Rafailov et al. (2024)

Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C.

Direct preference optimization: Your language model is secretly a reward model.

Advances in Neural Information Processing Systems, 36, 2024.

  • Raffel et al. (2020)

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J.

Exploring the limits of transfer learning with a unified text-to-text transformer.

Journal of machine learning research, 21(140):1–67, 2020.

  • Rajpurkar et al. (2018)

Rajpurkar, P., Jia, R., and Liang, P.

Know what you don’t know: Unanswerable questions for squad.

arXiv preprint arXiv:1806.03822, 2018.

  • Reddy et al. (2019)

Reddy, S., Chen, D., and Manning, C. D.

Coqa: A conversational question answering challenge.

Transactions of the Association for Computational Linguistics, 7:249–266, 2019.

  • Roberts et al. (2020)

Roberts, A., Raffel, C., and Shazeer, N.

How much knowledge can you pack into the parameters of a language model?

arXiv preprint arXiv:2002.08910, 2020.

  • Rolnick et al. (2017)

Rolnick, D., Veit, A., Belongie, S., and Shavit, N.

Deep learning is robust to massive label noise. arxiv 2017.

arXiv preprint arXiv:1705.10694, 2017.

  • Roziere et al. (2023)

Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al.

Code llama: Open foundation models for code.

arXiv preprint arXiv:2308.12950, 2023.

  • Sakaguchi et al. (2019)

Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y.

Winogrande: An adversarial winograd schema challenge at scale, 2019.

  • Sanh et al. (2021)

Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al.

Multitask prompted training enables zero-shot task generalization.

arXiv preprint arXiv:2110.08207, 2021.

  • Shao et al. (2024)

Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.

Deepseekmath: Pushing the limits of mathematical reasoning in open language models.

arXiv preprint arXiv:2402.03300, 2024.

  • Shu et al. (2024)

Shu, L., Luo, L., Hoskere, J., Zhu, Y., Liu, Y., Tong, S., Chen, J., and Meng, L.

Rewritelm: An instruction-tuned large language model for text rewriting.

In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18970–18980, 2024.

  • Singer et al. (2024)

Singer, P., Pfeiffer, P., Babakhin, Y., Jeblick, M., Dhankhar, N., Fodor, G., and Ambati, S. S.

H2o-danube-1.8 b technical report.

arXiv preprint arXiv:2401.16818, 2024.

  • Soboleva et al. (2023)

Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N.

Slimpajama: A 627b token cleaned and deduplicated version of redpajama, June 2023.

URL https://huggingface.co/datasets/cerebras/SlimPajama-627B.

  • Soldaini et al. (2024)

Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A. H., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Ravichander, A., Richardson, K., Shen, Z., Strubell, E., Subramani, N., Tafjord, O., Walsh, P., Zettlemoyer, L., Smith, N. A., Hajishirzi, H., Beltagy, I., Groeneveld, D., Dodge, J., and Lo, K.

Dolma: an open corpus of three trillion tokens for language model pretraining research, 2024.

  • Talmor et al. (2019)

Talmor, A., Herzig, J., Lourie, N., and Berant, J.

Commonsenseqa: A question answering challenge targeting commonsense knowledge.

In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4149–4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

doi: 10.18653/v1/N19-1421.

URL https://aclanthology.org/N19-1421.

  • Taylor et al. (2022)

Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R.

Galactica: A large language model for science.

arXiv preprint arXiv:2211.09085, 2022.

  • Team et al. (2024)

Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., et al.

Gemma 2: Improving open language models at a practical size, 2024.

URL https://arxiv. org/abs/2408.00118, 1(3), 2024.

  • Teknium (2023a)

Teknium.

Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023a.

URL https://huggingface.co/datasets/teknium/OpenHermes-2.5.

  • Teknium (2023b)

Teknium.

Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023b.

URL https://huggingface.co/datasets/teknium/OpenHermes-2.5.

  • Touvron et al. (2023)

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G.

Llama: Open and efficient foundation language models, 2023.

  • Vaswani et al. (2017)

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I.

Attention is all you need.

In Advances in Neural Information Processing Systems, 2017.

  • Wan et al. (2023)

Wan, F., Huang, X., Yang, T., Quan, X., Bi, W., and Shi, S.

Explore-instruct: Enhancing domain-specific instruction coverage through active exploration.

arXiv preprint arXiv:2310.09168, 2023.

  • Wang et al. (2024a)

Wang, H., Lin, Y., Xiong, W., Yang, R., Diao, S., Qiu, S., Zhao, H., and Zhang, T.

Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards.

In ACL, 2024a.

  • Wang et al. (2024b)

Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T.

Interpretable preferences via multi-objective reward modeling and mixture-of-experts.

In EMNLP, 2024b.

  • Wang et al. (2022)

Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H.

Self-instruct: Aligning language models with self-generated instructions.

arXiv preprint arXiv:2212.10560, 2022.

  • Wang et al. (2024c)

Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al.

Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.

arXiv preprint arXiv:2406.01574, 2024c.

  • (105)

Wang, Z., Li, X., Xia, R., and Liu, P.

Mathpile: A billion-token-scale pretraining corpus for math.

In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

  • Wei et al. (2021)

Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V.

Finetuned language models are zero-shot learners.

arXiv preprint arXiv:2109.01652, 2021.

  • Wei et al. (2024a)

Wei, Y., Cassano, F., Liu, J., Ding, Y., Jain, N., Mueller, Z., de Vries, H., von Werra, L., Guha, A., and Zhang, L.

Selfcodealign: Self-alignment for code generation.

arXiv preprint arXiv:2410.24198, 2024a.

  • Wei et al. (2024b)

Wei, Y., Han, H., and Samdani, R.

Arctic-snowcoder: Demystifying high-quality data in code pretraining.

arXiv preprint arXiv:2409.02326, 2024b.

  • Xu et al. (2024)

Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y.

Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing.

arXiv preprint arXiv:2406.08464, 2024.

  • Yang et al. (2024a)

Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K.-Y., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Fan, Y., Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Cui, Z., Zhang, Z., and Fan, Z.-W.

Qwen2 technical report.

ArXiv, abs/2407.10671, 2024a.

URL https://api.semanticscholar.org/CorpusID:271212307.

  • Yang et al. (2024b)

Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al.

Qwen2. 5 technical report.

arXiv preprint arXiv:2412.15115, 2024b.

  • Yang et al. (2024c)

Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al.

Qwen2.5-math technical report: Toward mathematical expert model via self-improvement.

arXiv preprint arXiv:2409.12122, 2024c.

  • Yen et al. (2024)

Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., and Chen, D.

Helmet: How to evaluate long-context language models effectively and thoroughly, 2024.

URL https://arxiv.org/abs/2410.02694.

  • Young et al. (2024)

Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., et al.

Yi: Open foundation models by 01.ai.

arXiv preprint arXiv:2403.04652, 2024.

  • Yu et al. (2023)

Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W.

Metamath: Bootstrap your own mathematical questions for large language models.

arXiv preprint arXiv:2309.12284, 2023.

  • Yuan et al. (2024)

Yuan, L., Cui, G., Wang, H., Ding, N., Wang, X., Deng, J., Shan, B., Chen, H., Xie, R., Lin, Y., et al.

Advancing llm reasoning generalists with preference trees.

arXiv preprint arXiv:2404.02078, 2024.

  • Zellers et al. (2019)

Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y.

Hellaswag: Can a machine really finish your sentence?

In Korhonen, A., Traum, D., and Màrquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics.

doi: 10.18653/v1/P19-1472.

URL https://aclanthology.org/P19-1472.

  • Zhai et al. (2022)

Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L.

Scaling vision transformers, 2022.

URL https://arxiv.org/abs/2106.04560.

  • Zhang et al. (2024)

Zhang, X., Zhang, Y., Long, D., Xie, W., Dai, Z., Tang, J., Lin, H., Yang, B., Xie, P., Huang, F., et al.

mgte: Generalized long-context text representation and reranking models for multilingual text retrieval.

arXiv preprint arXiv:2407.19669, 2024.

  • Zheng et al. (2023)

Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.

Judging llm-as-a-judge with mt-bench and chatbot arena.

Advances in Neural Information Processing Systems, 36:46595–46623, 2023.

  • Zhou et al. (2023)

Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L.

Instruction-following evaluation for large language models.

arXiv preprint arXiv:2311.07911, 2023.