References

本文为论文《DeepSeek-V3：强大高效的开源混合专家大模型》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

  • AI@Meta (2024a)

AI@Meta.

Llama 3 model card, 2024a.

URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.

  • AI@Meta (2024b)

AI@Meta.

Llama 3.1 model card, 2024b.

URL https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md.

  • Anthropic (2024)

Anthropic.

Claude 3.5 sonnet, 2024.

URL https://www.anthropic.com/news/claude-3-5-sonnet.

  • Austin et al. (2021)

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al.

Program synthesis with large language models.

arXiv preprint arXiv:2108.07732, 2021.

  • Bai et al. (2022)

Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al.

Constitutional AI: Harmlessness from AI feedback.

arXiv preprint arXiv:2212.08073, 2022.

  • Bai et al. (2024)

Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, J. Tang, and J. Li.

LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks.

arXiv preprint arXiv:2412.15204, 2024.

  • Bauer et al. (2014)

M. Bauer, S. Treichler, and A. Aiken.

Singe: leveraging warp specialization for high performance on GPUs.

In Proceedings of the 19th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP ’14, page 119–130, New York, NY, USA, 2014. Association for Computing Machinery.

ISBN 9781450326568.

10.1145/2555243.2555258.

URL https://doi.org/10.1145/2555243.2555258.

  • Bisk et al. (2020)

Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi.

PIQA: reasoning about physical commonsense in natural language.

In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020.

10.1609/aaai.v34i05.6239.

URL https://doi.org/10.1609/aaai.v34i05.6239.

  • Chen et al. (2021)

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba.

Evaluating large language models trained on code.

CoRR, abs/2107.03374, 2021.

URL https://arxiv.org/abs/2107.03374.

  • Clark et al. (2018)

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord.

Think you have solved question answering? try arc, the AI2 reasoning challenge.

CoRR, abs/1803.05457, 2018.

URL http://arxiv.org/abs/1803.05457.

  • Cobbe et al. (2021)

K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al.

Training verifiers to solve math word problems.

arXiv preprint arXiv:2110.14168, 2021.

  • Cui et al. (2019)

Y. Cui, T. Liu, W. Che, L. Xiao, Z. Chen, W. Ma, S. Wang, and G. Hu.

A span-extraction dataset for Chinese machine reading comprehension.

In K. Inui, J. Jiang, V. Ng, and X. Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5883–5889, Hong Kong, China, Nov. 2019. Association for Computational Linguistics.

10.18653/v1/D19-1600.

URL https://aclanthology.org/D19-1600.

  • Dai et al. (2024)

D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y. K. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang.

Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.

CoRR, abs/2401.06066, 2024.

URL https://doi.org/10.48550/arXiv.2401.06066.

  • DeepSeek-AI (2024a)

DeepSeek-AI.

Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence.

CoRR, abs/2406.11931, 2024a.

URL https://doi.org/10.48550/arXiv.2406.11931.

  • DeepSeek-AI (2024b)

DeepSeek-AI.

Deepseek LLM: scaling open-source language models with longtermism.

CoRR, abs/2401.02954, 2024b.

URL https://doi.org/10.48550/arXiv.2401.02954.

  • DeepSeek-AI (2024c)

DeepSeek-AI.

Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.

CoRR, abs/2405.04434, 2024c.

URL https://doi.org/10.48550/arXiv.2405.04434.

  • Dettmers et al. (2022)

T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer.

Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.

Advances in Neural Information Processing Systems, 35:30318–30332, 2022.

  • Ding et al. (2024)

H. Ding, Z. Wang, G. Paolini, V. Kumar, A. Deoras, D. Roth, and S. Soatto.

Fewer truncations improve language modeling.

arXiv preprint arXiv:2404.10830, 2024.

  • Dua et al. (2019)

D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner.

DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.

In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368–2378. Association for Computational Linguistics, 2019.

10.18653/V1/N19-1246.

URL https://doi.org/10.18653/v1/n19-1246.

  • Dubois et al. (2024)

Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto.

Length-controlled alpacaeval: A simple way to debias automatic evaluators.

arXiv preprint arXiv:2404.04475, 2024.

  • Fedus et al. (2021)

W. Fedus, B. Zoph, and N. Shazeer.

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.

CoRR, abs/2101.03961, 2021.

URL https://arxiv.org/abs/2101.03961.

  • Fishman et al. (2024)

M. Fishman, B. Chmiel, R. Banner, and D. Soudry.

Scaling FP8 training to trillion-token llms.

arXiv preprint arXiv:2409.12517, 2024.

  • Frantar et al. (2022)

E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh.

Gptq: Accurate post-training quantization for generative pre-trained transformers.

arXiv preprint arXiv:2210.17323, 2022.

  • Gao et al. (2020)

L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al.

The Pile: An 800GB dataset of diverse text for language modeling.

arXiv preprint arXiv:2101.00027, 2020.

  • Gema et al. (2024)

A. P. Gema, J. O. J. Leang, G. Hong, A. Devoto, A. C. M. Mancino, R. Saxena, X. He, Y. Zhao, X. Du, M. R. G. Madani, C. Barale, R. McHardy, J. Harris, J. Kaddour, E. van Krieken, and P. Minervini.

Are we done with mmlu?

CoRR, abs/2406.04127, 2024.

URL https://doi.org/10.48550/arXiv.2406.04127.

  • Gloeckle et al. (2024)

F. Gloeckle, B. Y. Idrissi, B. Rozière, D. Lopez-Paz, and G. Synnaeve.

Better & faster large language models via multi-token prediction.

In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.

URL https://openreview.net/forum?id=pEWAcejiU2.

  • Google (2024)

Google.

Our next-generation model: Gemini 1.5, 2024.

URL https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024.

  • Graham et al. (2016)

R. L. Graham, D. Bureddy, P. Lui, H. Rosenstock, G. Shainer, G. Bloch, D. Goldenerg, M. Dubman, S. Kotchubievsky, V. Koushnir, et al.

Scalable hierarchical aggregation protocol (SHArP): A hardware architecture for efficient data reduction.

In 2016 First International Workshop on Communication Optimizations in HPC (COMHPC), pages 1–10. IEEE, 2016.

  • Gu et al. (2024)

A. Gu, B. Rozière, H. Leather, A. Solar-Lezama, G. Synnaeve, and S. I. Wang.

Cruxeval: A benchmark for code reasoning, understanding and execution, 2024.

  • Guo et al. (2024)

D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, and W. Liang.

Deepseek-coder: When the large language model meets programming - the rise of code intelligence.

CoRR, abs/2401.14196, 2024.

URL https://doi.org/10.48550/arXiv.2401.14196.

  • Harlap et al. (2018)

A. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. Devanur, G. Ganger, and P. Gibbons.

Pipedream: Fast and efficient pipeline parallel dnn training, 2018.

URL https://arxiv.org/abs/1806.03377.

  • (32)

B. He, L. Noci, D. Paliotta, I. Schlag, and T. Hofmann.

Understanding and minimising outlier features in transformer training.

In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

  • He et al. (2024)

Y. He, S. Li, J. Liu, Y. Tan, W. Wang, H. Huang, X. Bu, H. Guo, C. Hu, B. Zheng, et al.

Chinese simpleqa: A chinese factuality evaluation for large language models.

arXiv preprint arXiv:2411.07140, 2024.

  • Hendrycks et al. (2020)

D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt.

Measuring massive multitask language understanding.

arXiv preprint arXiv:2009.03300, 2020.

  • Hendrycks et al. (2021)

D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.

Measuring mathematical problem solving with the math dataset.

arXiv preprint arXiv:2103.03874, 2021.

  • Huang et al. (2023)

Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al.

C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models.

arXiv preprint arXiv:2305.08322, 2023.

  • Jain et al. (2024)

N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica.

Livecodebench: Holistic and contamination free evaluation of large language models for code.

CoRR, abs/2403.07974, 2024.

URL https://doi.org/10.48550/arXiv.2403.07974.

  • Jiang et al. (2023)

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al.

Mistral 7b.

arXiv preprint arXiv:2310.06825, 2023.

  • Joshi et al. (2017)

M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer.

TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.

In R. Barzilay and M.-Y. Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada, July 2017. Association for Computational Linguistics.

10.18653/v1/P17-1147.

URL https://aclanthology.org/P17-1147.

  • Kalamkar et al. (2019)

D. Kalamkar, D. Mudigere, N. Mellempudi, D. Das, K. Banerjee, S. Avancha, D. T. Vooturi, N. Jammalamadaka, J. Huang, H. Yuen, et al.

A study of bfloat16 for deep learning training.

arXiv preprint arXiv:1905.12322, 2019.

  • Krishna et al. (2024)

S. Krishna, K. Krishna, A. Mohananey, S. Schwarcz, A. Stambler, S. Upadhyay, and M. Faruqui.

Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation.

CoRR, abs/2409.12941, 2024.

10.48550/ARXIV.2409.12941.

URL https://doi.org/10.48550/arXiv.2409.12941.

  • Kwiatkowski et al. (2019)

T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. P. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov.

Natural questions: a benchmark for question answering research.

Trans. Assoc. Comput. Linguistics, 7:452–466, 2019.

10.1162/tacl_a_00276.

URL https://doi.org/10.1162/tacl_a_00276.

  • Lai et al. (2017)

G. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy.

RACE: large-scale reading comprehension dataset from examinations.

In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017.

10.18653/V1/D17-1082.

URL https://doi.org/10.18653/v1/d17-1082.

  • Lambert et al. (2024)

N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, et al.

Rewardbench: Evaluating reward models for language modeling.

arXiv preprint arXiv:2403.13787, 2024.

  • Lepikhin et al. (2021)

D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.

Gshard: Scaling giant models with conditional computation and automatic sharding.

In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.

URL https://openreview.net/forum?id=qrwe7XHTmYb.

  • Leviathan et al. (2023)

Y. Leviathan, M. Kalman, and Y. Matias.

Fast inference from transformers via speculative decoding.

In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR, 2023.

URL https://proceedings.mlr.press/v202/leviathan23a.html.

  • Li et al. (2023)

H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin.

CMMLU: Measuring massive multitask language understanding in Chinese.

arXiv preprint arXiv:2306.09212, 2023.

  • Li and Hoefler (2021)

S. Li and T. Hoefler.

Chimera: efficiently training large-scale neural networks with bidirectional pipelines.

In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’21, page 1–14. ACM, Nov. 2021.

10.1145/3458817.3476145.

URL http://dx.doi.org/10.1145/3458817.3476145.

  • Li et al. (2024a)

T. Li, W.-L. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica.

From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.

arXiv preprint arXiv:2406.11939, 2024a.

  • Li et al. (2021)

W. Li, F. Qi, M. Sun, X. Yi, and J. Zhang.

Ccpm: A chinese classical poetry matching dataset, 2021.

  • Li et al. (2024b)

Y. Li, F. Wei, C. Zhang, and H. Zhang.

EAGLE: speculative sampling requires rethinking feature uncertainty.

In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024b.

URL https://openreview.net/forum?id=1NdN7eXyb4.

  • Lin (2024)

B. Y. Lin.

ZeroEval: A Unified Framework for Evaluating Language Models, July 2024.

URL https://github.com/WildEval/ZeroEval.

  • Loshchilov and Hutter (2017)

I. Loshchilov and F. Hutter.

Decoupled weight decay regularization.

arXiv preprint arXiv:1711.05101, 2017.

  • Lundberg (2023)

S. Lundberg.

The art of prompt design: Prompt boundaries and token healing, 2023.

URL https://towardsdatascience.com/the-art-of-prompt-design-prompt-boundaries-and-token-healing-3b2448b0be38.

  • Luo et al. (2024)

Y. Luo, Z. Zhang, R. Wu, H. Liu, Y. Jin, K. Zheng, M. Wang, Z. He, G. Hu, L. Chen, et al.

Ascend HiFloat8 format for deep learning.

arXiv preprint arXiv:2409.16626, 2024.

  • MAA (2024)

MAA.

American invitational mathematics examination - aime.

In American Invitational Mathematics Examination - AIME 2024, February 2024.

URL https://maa.org/math-competitions/american-invitational-mathematics-examination-aime.

  • Micikevicius et al. (2022)

P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Heinecke, P. Judd, J. Kamalu, et al.

FP8 formats for deep learning.

arXiv preprint arXiv:2209.05433, 2022.

  • Mistral (2024)

Mistral.

Cheaper, better, faster, stronger: Continuing to push the frontier of ai and making it accessible to all, 2024.

URL https://mistral.ai/news/mixtral-8x22b.

  • Narang et al. (2017)

S. Narang, G. Diamos, E. Elsen, P. Micikevicius, J. Alben, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, et al.

Mixed precision training.

In Int. Conf. on Learning Representation, 2017.

  • Noune et al. (2022)

B. Noune, P. Jones, D. Justus, D. Masters, and C. Luschi.

8-bit numerical formats for deep neural networks.

arXiv preprint arXiv:2206.02915, 2022.

  • NVIDIA (2022)

NVIDIA.

Improving network performance of HPC systems using NVIDIA Magnum IO NVSHMEM and GPUDirect Async.

https://developer.nvidia.com/blog/improving-network-performance-of-hpc-systems-using-nvidia-magnum-io-nvshmem-and-gpudirect-async, 2022.

  • NVIDIA (2024a)

NVIDIA.

Blackwell architecture.

https://www.nvidia.com/en-us/data-center/technologies/blackwell-architecture/, 2024a.

  • NVIDIA (2024b)

NVIDIA.

TransformerEngine, 2024b.

URL https://github.com/NVIDIA/TransformerEngine.

Accessed: 2024-11-19.

  • OpenAI (2024a)

OpenAI.

Hello GPT-4o, 2024a.

URL https://openai.com/index/hello-gpt-4o/.

  • OpenAI (2024b)

OpenAI.

Multilingual massive multitask language understanding (mmmlu), 2024b.

URL https://huggingface.co/datasets/openai/MMMLU.

  • OpenAI (2024c)

OpenAI.

Introducing SimpleQA, 2024c.

URL https://openai.com/index/introducing-simpleqa/.

  • OpenAI (2024d)

OpenAI.

Introducing SWE-bench verified we’re releasing a human-validated subset of swe-bench that more, 2024d.

URL https://openai.com/index/introducing-swe-bench-verified/.

  • Peng et al. (2023a)

B. Peng, J. Quesnelle, H. Fan, and E. Shippole.

Yarn: Efficient context window extension of large language models.

arXiv preprint arXiv:2309.00071, 2023a.

  • Peng et al. (2023b)

H. Peng, K. Wu, Y. Wei, G. Zhao, Y. Yang, Z. Liu, Y. Xiong, Z. Yang, B. Ni, J. Hu, et al.

FP8-LM: Training FP8 large language models.

arXiv preprint arXiv:2310.18313, 2023b.

  • Qi et al. (2023a)

P. Qi, X. Wan, G. Huang, and M. Lin.

Zero bubble pipeline parallelism.

arXiv preprint arXiv:2401.10241, 2023a.

  • Qi et al. (2023b)

P. Qi, X. Wan, G. Huang, and M. Lin.

Zero bubble pipeline parallelism, 2023b.

URL https://arxiv.org/abs/2401.10241.

  • Qwen (2023)

Qwen.

Qwen technical report.

arXiv preprint arXiv:2309.16609, 2023.

  • Qwen (2024a)

Qwen.

Introducing Qwen1.5, 2024a.

URL https://qwenlm.github.io/blog/qwen1.5.

  • Qwen (2024b)

Qwen.

Qwen2.5: A party of foundation models, 2024b.

URL https://qwenlm.github.io/blog/qwen2.5.

  • Rajbhandari et al. (2020)

S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He.

Zero: Memory optimizations toward training trillion parameter models.

In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020.

  • Rein et al. (2023)

D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman.

GPQA: A graduate-level google-proof q&a benchmark.

arXiv preprint arXiv:2311.12022, 2023.

  • Rouhani et al. (2023a)

B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolf, et al.

Microscaling data formats for deep learning.

arXiv preprint arXiv:2310.10537, 2023a.

  • Rouhani et al. (2023b)

B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolf, et al.

Microscaling data formats for deep learning.

arXiv preprint arXiv:2310.10537, 2023b.

  • Sakaguchi et al. (2019)

K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi.

Winogrande: An adversarial winograd schema challenge at scale, 2019.

  • Shao et al. (2024)

Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo.

Deepseekmath: Pushing the limits of mathematical reasoning in open language models.

arXiv preprint arXiv:2402.03300, 2024.

  • Shazeer et al. (2017)

N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean.

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.

In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017.

URL https://openreview.net/forum?id=B1ckMDqlg.

  • Shi et al. (2023)

F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, D. Das, and J. Wei.

Language models are multilingual chain-of-thought reasoners.

In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.

URL https://openreview.net/forum?id=fR3wGCk-IXp.

  • Shibata et al. (1999)

Y. Shibata, T. Kida, S. Fukamachi, M. Takeda, A. Shinohara, T. Shinohara, and S. Arikawa.

Byte pair encoding: A text compression scheme that accelerates pattern matching.

1999.

  • Su et al. (2024)

J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu.

Roformer: Enhanced transformer with rotary position embedding.

Neurocomputing, 568:127063, 2024.

  • Sun et al. (2019a)

K. Sun, D. Yu, D. Yu, and C. Cardie.

Investigating prior knowledge for challenging chinese machine reading comprehension, 2019a.

  • Sun et al. (2024)

M. Sun, X. Chen, J. Z. Kolter, and Z. Liu.

Massive activations in large language models.

arXiv preprint arXiv:2402.17762, 2024.

  • Sun et al. (2019b)

X. Sun, J. Choi, C.-Y. Chen, N. Wang, S. Venkataramani, V. V. Srinivasan, X. Cui, W. Zhang, and K. Gopalakrishnan.

Hybrid 8-bit floating point (HFP8) training and inference for deep neural networks.

Advances in neural information processing systems, 32, 2019b.

  • Suzgun et al. (2022)

M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al.

Challenging big-bench tasks and whether chain-of-thought can solve them.

arXiv preprint arXiv:2210.09261, 2022.

  • Thakkar et al. (2023)

V. Thakkar, P. Ramani, C. Cecka, A. Shivam, H. Lu, E. Yan, J. Kosaian, M. Hoemmen, H. Wu, A. Kerr, M. Nicely, D. Merrill, D. Blasig, F. Qiao, P. Majcher, P. Springer, M. Hohnerbach, J. Wang, and M. Gupta.

CUTLASS, Jan. 2023.

URL https://github.com/NVIDIA/cutlass.

  • Touvron et al. (2023a)

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.

LLaMA: Open and efficient foundation language models.

arXiv preprint arXiv:2302.13971, 2023a.

  • Touvron et al. (2023b)

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom.

Llama 2: Open foundation and fine-tuned chat models.

CoRR, abs/2307.09288, 2023b.

10.48550/arXiv.2307.09288.

URL https://doi.org/10.48550/arXiv.2307.09288.

  • Vaswani et al. (2017)

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.

Attention is all you need.

Advances in neural information processing systems, 30, 2017.

  • Wang et al. (2024a)

L. Wang, H. Gao, C. Zhao, X. Sun, and D. Dai.

Auxiliary-loss-free load balancing strategy for mixture-of-experts.

CoRR, abs/2408.15664, 2024a.

URL https://doi.org/10.48550/arXiv.2408.15664.

  • Wang et al. (2024b)

Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen.

Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.

CoRR, abs/2406.01574, 2024b.

URL https://doi.org/10.48550/arXiv.2406.01574.

  • Wei et al. (2023)

T. Wei, J. Luan, W. Liu, S. Dong, and B. Wang.

Cmath: Can your language model pass chinese elementary school math test?, 2023.

  • Wortsman et al. (2023)

M. Wortsman, T. Dettmers, L. Zettlemoyer, A. Morcos, A. Farhadi, and L. Schmidt.

Stable and low-precision training for large-scale vision-language models.

Advances in Neural Information Processing Systems, 36:10271–10298, 2023.

  • Xi et al. (2023)

H. Xi, C. Li, J. Chen, and J. Zhu.

Training transformers with 4-bit integers.

Advances in Neural Information Processing Systems, 36:49146–49168, 2023.

  • Xia et al. (2024)

C. S. Xia, Y. Deng, S. Dunn, and L. Zhang.

Agentless: Demystifying llm-based software engineering agents.

arXiv preprint, 2024.

  • Xia et al. (2023)

H. Xia, T. Ge, P. Wang, S. Chen, F. Wei, and Z. Sui.

Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation.

In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 3909–3925. Association for Computational Linguistics, 2023.

URL https://doi.org/10.18653/v1/2023.findings-emnlp.257.

  • Xiao et al. (2023)

G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han.

Smoothquant: Accurate and efficient post-training quantization for large language models.

In International Conference on Machine Learning, pages 38087–38099. PMLR, 2023.

  • Xu et al. (2020)

L. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan.

CLUE: A chinese language understanding evaluation benchmark.

In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020.

10.18653/V1/2020.COLING-MAIN.419.

URL https://doi.org/10.18653/v1/2020.coling-main.419.

  • Zellers et al. (2019)

R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi.

HellaSwag: Can a machine really finish your sentence?

In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019.

10.18653/v1/p19-1472.

URL https://doi.org/10.18653/v1/p19-1472.

  • Zhong et al. (2023)

W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan.

AGIEval: A human-centric benchmark for evaluating foundation models.

CoRR, abs/2304.06364, 2023.

10.48550/arXiv.2304.06364.

URL https://doi.org/10.48550/arXiv.2304.06364.

  • Zhou et al. (2023)

J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou.

Instruction-following evaluation for large language models.

arXiv preprint arXiv:2311.07911, 2023.