References

本文为论文《SmolLM2：紧凑高效的小语言模型家族》「中文全文翻译」板块的对应章节，与「论文原文」板块同一章节一一对应，层级保持一致。

  • (1) Claudebot documentation.

https://darkvisitors.com/agents/claudebot.

Accessed: 2024-06-05.

  • (2) Common crawl.

https://commoncrawl.org/.

Accessed: 2024-06-05.

  • (3) Openai gptbot documentation.

https://platform.openai.com/docs/gptbot.

Accessed: 2024-06-05.

  • Abdin et al. (2024a)

Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al.

Phi-3 technical report: A highly capable language model locally on your phone.

arXiv 预印本 arXiv:2404.14219, 2024a.

  • Abdin et al. (2024b)

Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al.

Phi-4 technical report.

arXiv 预印本 arXiv:2412.08905, 2024b.

  • Achiam et al. (2023)

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.

Gpt-4 technical report.

arXiv 预印本 arXiv:2303.08774, 2023.

  • Ai2 (2024)

Ai2.

Olmo 2: The best fully open language model to date.

https://allenai.org/blog/olmo2, 2024.

Blog post.

  • AI@Meta (2024a)

AI@Meta.

Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, 2024a.

URL https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/.

  • AI@Meta (2024b)

AI@Meta.

Llama 3.2 model card, 2024b.

URL https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md.

  • Allal et al. (2024)

Allal, L. B., Lozhkov, A., Bakouch, E., von Werra, L., and Wolf, T.

Smollm - blazingly fast and remarkably powerful, 2024.

  • Almazrouei et al. (2023)

Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Goffinet, É., Hesslow, D., Launay, J., Malartic, Q., et al.

The falcon series of open language models.

arXiv 预印本 arXiv:2311.16867, 2023.

  • Argilla (2024)

Argilla.

Finepersonas-v0.1 dataset.

https://huggingface.co/datasets/argilla/FinePersonas-v0.1, 2024.

Available on Hugging Face Datasets.

  • Aryabumi et al. (2024)

Aryabumi, V., Su, Y., Ma, R., Morisot, A., Zhang, I., Locatelli, A., Fadaee, M., Üstün, A., and Hooker, S.

To code, or not to code? exploring impact of code in pre-training.

arXiv 预印本 arXiv:2408.10914, 2024.

  • Azerbayev et al. (2023)

Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S.

Llemma: An open language model for mathematics.

arXiv 预印本 arXiv:2310.10631, 2023.

  • Bai et al. (2023)

Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.

Qwen technical report.

arXiv 预印本 arXiv:2309.16609, 2023.

  • Bai et al. (2022)

Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al.

Constitutional ai: Harmlessness from ai feedback.

arXiv 预印本 arXiv:2212.08073, 2022.

  • Bai et al. (2024)

Bai, Y., Lv, X., Zhang, J., He, Y., Qi, J., Hou, L., Tang, J., Dong, Y., and Li, J.

Longalign: A recipe for long context alignment of large language models.

arXiv 预印本 arXiv:2401.18058, 2024.

  • Bartolomé Del Canto et al. (2024)

Bartolomé Del Canto, Á., Martín Blázquez, G., Piqueres Lajarín, A., and Vila Suero, D.

Distilabel: An AI feedback (AIF) framework for building datasets with and for LLMs.

https://github.com/argilla-io/distilabel, 2024.

  • Bisk et al. (2019)

Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y.

Piqa: Reasoning about physical commonsense in natural language, 2019.

  • Blakeney et al. (2024)

Blakeney, C., Paul, M., Larsen, B. W., Owen, S., and Frankle, J.

Does your data spark joy? performance gains from domain upsampling at the end of training.

arXiv 预印本 arXiv:2406.03476, 2024.

  • Broder (1997)

Broder, A. Z.

On the resemblance and containment of documents.

Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171), pp. 21–29, 1997.

  • Brown et al. (2020)

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.

Language models are few-shot learners.

arXiv 预印本 arXiv:2005.14165, 2020.

  • Cassano et al. (2022)

Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al.

Multipl-e: A scalable and extensible approach to benchmarking neural code generation.

arXiv 预印本 arXiv:2208.08227, 2022.

  • Chan et al. (2024)

Chan, X., Wang, X., Yu, D., Mi, H., and Yu, D.

Scaling synthetic data creation with 1,000,000,000 personas, 2024.

URL https://arxiv.org/abs/2406.20094.

  • Chen et al. (2021)

Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al.

Evaluating large language models trained on code.

arXiv 预印本 arXiv:2107.03374, 2021.

  • Chowdhery et al. (2023)

Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al.

Palm: Scaling language modeling with pathways.

Journal of Machine Learning Research, 24(240):1–113, 2023.

  • Clark (2019)

Clark, K.

What does bert look at? an analysis of bert’s attention.

arXiv 预印本 arXiv:1906.04341, 2019.

  • Clark et al. (2018)

Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O.

Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.

  • Cobbe et al. (2021)

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.

Training verifiers to solve math word problems.

arXiv 预印本 arXiv:2110.14168, 2021.

  • Computations (2024)

Computations, C.

Systemchat-2.0.

https://huggingface.co/datasets/cognitivecomputations/SystemChat-2.0, 2024.

  • Cui et al. (2024)

Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., et al.

Ultrafeedback: Boosting language models with scaled ai feedback.

In Forty-first International Conference on Machine Learning, 2024.

  • Daniele & Suphavadeeprasit (2023)

Daniele, L. and Suphavadeeprasit.

Amplify-instruct: Synthetically generated diverse multi-turn conversations for efficient llm training.

arXiv 预印本 arXiv:(coming soon), 2023.

URL https://huggingface.co/datasets/LDJnr/Capybara.

  • de Vries (2023)

de Vries, H.

Go smol or go home.

https://www.harmdevries.com/post/model-size-vs-compute-overhead/, 2023.

  • Ding et al. (2023)

Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B.

Enhancing chat language models by scaling high-quality instructional conversations.

arXiv 预印本 arXiv:2305.14233, 2023.

  • Du et al. (2024)

Du, Z., Zeng, A., Dong, Y., and Tang, J.

Understanding emergent abilities of language models from the loss perspective.

arXiv 预印本 arXiv:2403.15796, 2024.

  • Dua et al. (2019)

Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M.

Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs.

arXiv 预印本 arXiv:1903.00161, 2019.

  • Dubey et al. (2024)

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.

The llama 3 herd of models.

arXiv 预印本 arXiv:2407.21783, 2024.

  • Face (2024)

Face, H.

Everyday conversations for llms.

https://huggingface.co/datasets/HuggingFaceTB/everyday-conversations-llama3.1-2k, 2024.

  • Gao et al. (2024)

Gao, T., Wettig, A., Yen, H., and Chen, D.

How to train long-context language models (effectively), 2024.

URL https://arxiv.org/abs/2410.02660.

  • Ge et al. (2024)

Ge, T., Chan, X., Wang, X., Yu, D., Mi, H., and Yu, D.

Scaling synthetic data creation with 1,000,000,000 personas.

arXiv 预印本 arXiv:2406.20094, 2024.

  • Groeneveld et al. (2024)

Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al.

Olmo: Accelerating the science of language models.

arXiv 预印本 arXiv:2402.00838, 2024.

  • Gu et al. (2024)

Gu, Y., Tafjord, O., Kuehl, B., Haddad, D., Dodge, J., and Hajishirzi, H.

Olmes: A standard for language model evaluations.

arXiv 预印本 arXiv:2406.08446, 2024.

  • Gunter et al. (2024)

Gunter, T., Wang, Z., Wang, C., Pang, R., Narayanan, A., Zhang, A., Zhang, B., Chen, C., Chiu, C.-C., Qiu, D., et al.

Apple intelligence foundation language models.

arXiv 预印本 arXiv:2407.21075, 2024.

  • Hägele et al. (2024)

Hägele, A., Bakouch, E., Kosson, A., Allal, L. B., Von Werra, L., and Jaggi, M.

Scaling laws and compute-optimal training beyond fixed training durations.

arXiv 预印本 arXiv:2405.18392, 2024.

  • Han et al. (2024)

Han, X., Jian, Y., Hu, X., Liu, H., Wang, Y., Fan, Q., Ai, Y., Huang, H., He, R., Yang, Z., et al.

Infimm-webmath-40b: Advancing multimodal pre-training for enhanced mathematical reasoning.

arXiv 预印本 arXiv:2409.12568, 2024.

  • Hendrycks et al. (2021)

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J.

Measuring massive multitask language understanding, 2021.

  • Hoffmann et al. (2022)

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al.

Training compute-optimal large language models.

arXiv 预印本 arXiv:2203.15556, 2022.

  • Hu et al. (2024)

Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., Zhang, X., Thai, Z. L., Zhang, K., Wang, C., Yao, Y., Zhao, C., Zhou, J., Cai, J., Zhai, Z., Ding, N., Jia, C., Zeng, G., Li, D., Liu, Z., and Sun, M.

Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024.

URL https://arxiv.org/abs/2404.06395.

  • Ivison et al. (2024)

Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., and Hajishirzi, H.

Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback.

arXiv 预印本 arXiv:2406.09279, 2024.

  • Joshi et al. (2017)

Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L.

Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.

arXiv 预印本 arXiv:1705.03551, 2017.

  • Joulin et al. (2016a)

Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., and Mikolov, T.

Fasttext.zip: Compressing text classification models.

arXiv 预印本 arXiv:1612.03651, 2016a.

  • Joulin et al. (2016b)

Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T.

Bag of tricks for efficient text classification.

arXiv 预印本 arXiv:1607.01759, 2016b.

  • Kamradt (2024)

Kamradt, G.

Needle in a haystack - pressure testing llms.

https://github.com/gkamradt/LLMTestNeedleInAHaystack, 2024.

  • Kocetkov et al. (2022)

Kocetkov, D., Li, R., Allal, L. B., Li, J., Mou, C., Ferrandis, C. M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., et al.

The stack: 3 tb of permissively licensed source code.

arXiv 预印本 arXiv:2211.15533, 2022.

  • Kong et al. (2024)

Kong, X., Gunter, T., and Pang, R.

Large language model-guided document selection.

arXiv 预印本 arXiv:2406.04638, 2024.

  • Kwiatkowski et al. (2019)

Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al.

Natural questions: a benchmark for question answering research.

Transactions of the Association for Computational Linguistics, 7, 2019.

  • (57)

Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K. R., Bishop, C., Hall, E., Carbune, V., Rastogi, A., et al.

Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback.

In Forty-first International Conference on Machine Learning.

  • Lewkowycz et al. (2022)

Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al.

Solving quantitative reasoning problems with language models.

Advances in Neural Information Processing Systems, 35:3843–3857, 2022.

  • Li et al. (2024a)

Li, C., Yuan, Z., Yuan, H., Dong, G., Lu, K., Wu, J., Tan, C., Wang, X., and Zhou, C.

Mugglemath: Assessing the impact of query and response augmentation on math reasoning.

In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10230–10258, 2024a.

  • Li et al. (2024b)

Li, J., Beeching, E., Tunstall, L., Lipkin, B., Soletskyi, R., Huang, S. C., Rasul, K., Yu, L., Jiang, A., Shen, Z., Qin, Z., Dong, B., Zhou, L., Fleureau, Y., Lample, G., and Polu, S.

Numinamath.

https://huggingface.co/AI-MO/NuminaMath-CoT, 2024b.

  • Li et al. (2024c)

Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha, E., Keh, S., Arora, K., et al.

Datacomp-LM: In search of the next generation of training sets for language models.

arXiv 预印本 arXiv:2406.11794, 2024c.

  • Li et al. (2023a)

Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al.

Starcoder: may the source be with you!

arXiv 预印本 arXiv:2305.06161, 2023a.

  • Li et al. (2023b)

Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T.

Textbooks are all you need ii: phi-1.5 technical report.

arXiv 预印本 arXiv:2309.05463, 2023b.

  • Li et al. (2023c)

Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., and Zhang, M.

Towards general text embeddings with multi-stage contrastive learning.

arXiv 预印本 arXiv:2308.03281, 2023c.

  • Liu et al. (2024)

Liu, Z., Hoang, T., Zhang, J., Zhu, M., Lan, T., Kokane, S., Tan, J., Yao, W., Liu, Z., Feng, Y., et al.

Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets.

arXiv 预印本 arXiv:2406.18518, 2024.

  • Loshchilov & Hutter (2016)

Loshchilov, I. and Hutter, F.

SGDR: Stochastic gradient descent with warm restarts.

arXiv 预印本 arXiv:1608.03983, 2016.

  • Lozhkov et al. (2024)

Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al.

Starcoder 2 and the stack v2: The next generation.

arXiv 预印本 arXiv:2402.19173, 2024.

  • Lv et al. (2023)

Lv, K., Zhang, W., and Shen, H.

Supervised fine-tuning and direct preference optimization on intel gaudi2.

https://medium.com/intel-analytics-software/a1197d8a3cd3, 2023.

Intel Corporation.

  • Mihaylov et al. (2018)

Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A.

Can a suit of armor conduct electricity? a new dataset for open book question answering.

In EMNLP, 2018.

  • Mishra et al. (2021)

Mishra, S., Khashabi, D., Baral, C., and Hajishirzi, H.

Cross-task generalization via natural language crowdsourcing instructions.

arXiv 预印本 arXiv:2104.08773, 2021.

  • MosaicML (2024)

MosaicML.

Llm foundry - jeopardy dataset, 2024.

URL https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/local_data/world_knowledge/jeopardy_all.jsonl.

Accessed: 2024-11-10.

  • Muennighoff et al. (2023)

Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., and Raffel, C.

Scaling data-constrained language models, 2023.

  • Ouyang et al. (2022)

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.

Training language models to follow instructions with human feedback.

Advances in neural information processing systems, 35:27730–27744, 2022.

  • Paster et al. (2023)

Paster, K., Santos, M. D., Azerbayev, Z., and Ba, J.

Openwebmath: An open dataset of high-quality mathematical web text.

arXiv 预印本 arXiv:2310.06786, 2023.

  • Penedo et al. (2024a)

Penedo, G., Kydlícek, H., Allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., von Werra, L., and Wolf, T.

The fineweb datasets: Decanting the web for the finest text data at scale.

ArXiv, abs/2406.17557, 2024a.

URL https://api.semanticscholar.org/CorpusID:270711474.

  • Penedo et al. (2024b)

Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Alobeidli, H., Cappelli, A., Pannier, B., Almazrouei, E., and Launay, J.

The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data only.

In Advances in Neural Information Processing Systems, 2024b.

  • Petroni et al. (2019)

Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., and Riedel, S.

Language models as knowledge bases?

arXiv 预印本 arXiv:1909.01066, 2019.

  • Radford et al. (2019)

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I.

Language models are unsupervised multitask learners.

2019.

  • Rafailov et al. (2024)

Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C.

Direct preference optimization: Your language model is secretly a reward model.

Advances in Neural Information Processing Systems, 36, 2024.

  • Raffel et al. (2020)

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J.

Exploring the limits of transfer learning with a unified text-to-text transformer.

Journal of machine learning research, 21(140):1–67, 2020.

  • Rajpurkar et al. (2018)

Rajpurkar, P., Jia, R., and Liang, P.

Know what you don’t know: Unanswerable questions for squad.

arXiv 预印本 arXiv:1806.03822, 2018.

  • Reddy et al. (2019)

Reddy, S., Chen, D., and Manning, C. D.

Coqa: A conversational question answering challenge.

Transactions of the Association for Computational Linguistics, 7:249–266, 2019.

  • Roberts et al. (2020)

Roberts, A., Raffel, C., and Shazeer, N.

How much knowledge can you pack into the parameters of a language model?

arXiv 预印本 arXiv:2002.08910, 2020.

  • Rolnick et al. (2017)

Rolnick, D., Veit, A., Belongie, S., and Shavit, N.

Deep learning is robust to massive label noise. arxiv 2017.

arXiv 预印本 arXiv:1705.10694, 2017.

  • Roziere et al. (2023)

Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al.

Code llama: Open foundation models for code.

arXiv 预印本 arXiv:2308.12950, 2023.

  • Sakaguchi et al. (2019)

Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y.

Winogrande: An adversarial winograd schema challenge at scale, 2019.

  • Sanh et al. (2021)

Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al.

Multitask prompted training enables zero-shot task generalization.

arXiv 预印本 arXiv:2110.08207, 2021.

  • Shao et al. (2024)

Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.

Deepseekmath: Pushing the limits of mathematical reasoning in open language models.

arXiv 预印本 arXiv:2402.03300, 2024.

  • Shu et al. (2024)

Shu, L., Luo, L., Hoskere, J., Zhu, Y., Liu, Y., Tong, S., Chen, J., and Meng, L.

Rewritelm: An instruction-tuned large language model for text rewriting.

In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18970–18980, 2024.

  • Singer et al. (2024)

Singer, P., Pfeiffer, P., Babakhin, Y., Jeblick, M., Dhankhar, N., Fodor, G., and Ambati, S. S.

H2o-danube-1.8 b technical report.

arXiv 预印本 arXiv:2401.16818, 2024.

  • Soboleva et al. (2023)

Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N.

Slimpajama: A 627b token cleaned and deduplicated version of redpajama, June 2023.

URL https://huggingface.co/datasets/cerebras/SlimPajama-627B.

  • Soldaini et al. (2024)

Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A. H., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Ravichander, A., Richardson, K., Shen, Z., Strubell, E., Subramani, N., Tafjord, O., Walsh, P., Zettlemoyer, L., Smith, N. A., Hajishirzi, H., Beltagy, I., Groeneveld, D., Dodge, J., and Lo, K.

Dolma: an open corpus of three trillion tokens for language model pretraining research, 2024.

  • Talmor et al. (2019)

Talmor, A., Herzig, J., Lourie, N., and Berant, J.

Commonsenseqa: A question answering challenge targeting commonsense knowledge.

In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4149–4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

doi: 10.18653/v1/N19-1421.

URL https://aclanthology.org/N19-1421.

  • Taylor et al. (2022)

Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R.

Galactica: A large language model for science.

arXiv 预印本 arXiv:2211.09085, 2022.

  • Team et al. (2024)

Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., et al.

Gemma 2: Improving open language models at a practical size, 2024.

URL https://arxiv. org/abs/2408.00118, 1(3), 2024.

  • Teknium (2023a)

Teknium.

Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023a.

URL https://huggingface.co/datasets/teknium/OpenHermes-2.5.

  • Teknium (2023b)

Teknium.

Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023b.

URL https://huggingface.co/datasets/teknium/OpenHermes-2.5.

  • Touvron et al. (2023)

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G.

Llama: Open and efficient foundation language models, 2023.

  • Vaswani et al. (2017)

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I.

Attention is all you need.

In Advances in Neural Information Processing Systems, 2017.

  • Wan et al. (2023)

Wan, F., Huang, X., Yang, T., Quan, X., Bi, W., and Shi, S.

Explore-instruct: Enhancing domain-specific instruction coverage through active exploration.

arXiv 预印本 arXiv:2310.09168, 2023.

  • Wang et al. (2024a)

Wang, H., Lin, Y., Xiong, W., Yang, R., Diao, S., Qiu, S., Zhao, H., and Zhang, T.

Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards.

In ACL, 2024a.

  • Wang et al. (2024b)

Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T.

Interpretable preferences via multi-objective reward modeling and mixture-of-experts.

In EMNLP, 2024b.

  • Wang et al. (2022)

Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H.

Self-instruct: Aligning language models with self-generated instructions.

arXiv 预印本 arXiv:2212.10560, 2022.

  • Wang et al. (2024c)

Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al.

Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.

arXiv 预印本 arXiv:2406.01574, 2024c.

  • (105)

Wang, Z., Li, X., Xia, R., and Liu, P.

Mathpile: A billion-token-scale pretraining corpus for math.

In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

  • Wei et al. (2021)

Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V.

Finetuned language models are zero-shot learners.

arXiv 预印本 arXiv:2109.01652, 2021.

  • Wei et al. (2024a)

Wei, Y., Cassano, F., Liu, J., Ding, Y., Jain, N., Mueller, Z., de Vries, H., von Werra, L., Guha, A., and Zhang, L.

Selfcodealign: Self-alignment for code generation.

arXiv 预印本 arXiv:2410.24198, 2024a.

  • Wei et al. (2024b)

Wei, Y., Han, H., and Samdani, R.

Arctic-snowcoder: Demystifying high-quality data in code pretraining.

arXiv 预印本 arXiv:2409.02326, 2024b.

  • Xu et al. (2024)

Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y.

Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing.

arXiv 预印本 arXiv:2406.08464, 2024.

  • Yang et al. (2024a)

Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K.-Y., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Fan, Y., Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Cui, Z., Zhang, Z., and Fan, Z.-W.

Qwen2 technical report.

ArXiv, abs/2407.10671, 2024a.

URL https://api.semanticscholar.org/CorpusID:271212307.

  • Yang et al. (2024b)

Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al.

Qwen2. 5 technical report.

arXiv 预印本 arXiv:2412.15115, 2024b.

  • Yang et al. (2024c)

Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al.

Qwen2.5-math technical report: Toward mathematical expert model via self-improvement.

arXiv 预印本 arXiv:2409.12122, 2024c.

  • Yen et al. (2024)

Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., and Chen, D.

Helmet: How to evaluate long-context language models effectively and thoroughly, 2024.

URL https://arxiv.org/abs/2410.02694.

  • Young et al. (2024)

Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., et al.

Yi: Open foundation models by 01.ai.

arXiv 预印本 arXiv:2403.04652, 2024.

  • Yu et al. (2023)

Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W.

Metamath: Bootstrap your own mathematical questions for large language models.

arXiv 预印本 arXiv:2309.12284, 2023.

  • Yuan et al. (2024)

Yuan, L., Cui, G., Wang, H., Ding, N., Wang, X., Deng, J., Shan, B., Chen, H., Xie, R., Lin, Y., et al.

Advancing llm reasoning generalists with preference trees.

arXiv 预印本 arXiv:2404.02078, 2024.

  • Zellers et al. (2019)

Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y.

Hellaswag: Can a machine really finish your sentence?

In Korhonen, A., Traum, D., and Màrquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics.

doi: 10.18653/v1/P19-1472.

URL https://aclanthology.org/P19-1472.

  • Zhai et al. (2022)

Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L.

Scaling vision transformers, 2022.

URL https://arxiv.org/abs/2106.04560.

  • Zhang et al. (2024)

Zhang, X., Zhang, Y., Long, D., Xie, W., Dai, Z., Tang, J., Lin, H., Yang, B., Xie, P., Huang, F., et al.

mgte: Generalized long-context text representation and reranking models for multilingual text retrieval.

arXiv 预印本 arXiv:2407.19669, 2024.

  • Zheng et al. (2023)

Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.

Judging llm-as-a-judge with mt-bench and chatbot arena.

Advances in Neural Information Processing Systems, 36:46595–46623, 2023.

  • Zhou et al. (2023)

Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L.

Instruction-following evaluation for large language models.

arXiv 预印本 arXiv:2311.07911, 2023.