参考文献

本文为论文《DeepSeek-V3：强大高效的开源混合专家大模型》「中文全文翻译」板块的对应章节，与「论文原文」板块同一章节一一对应，层级保持一致。

  • AI@Meta (2024a)

AI@Meta.

Llama 3 模型卡片, 2024a.

URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.

  • AI@Meta (2024b)

AI@Meta.

Llama 3.1 模型卡片, 2024b.

URL https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md.

  • Anthropic (2024)

Anthropic.

Claude 3.5 sonnet, 2024.

URL https://www.anthropic.com/news/claude-3-5-sonnet.

  • Austin et al. (2021)

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al.

使用大语言模型进行程序合成。

arXiv preprint arXiv:2108.07732, 2021.

  • Bai et al. (2022)

Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al.

宪法式 AI：来自 AI 反馈的无害性。

arXiv preprint arXiv:2212.08073, 2022.

  • Bai et al. (2024)

Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, J. Tang, and J. Li.

LongBench v2：面向真实长上下文多任务上的更深层次理解与推理。

arXiv preprint arXiv:2412.15204, 2024.

  • Bauer et al. (2014)

M. Bauer, S. Treichler, and A. Aiken.

Singe：利用 warp 专门化在 GPU 上实现高性能。

In Proceedings of the 19th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP ’14, page 119–130, New York, NY, USA, 2014. Association for Computing Machinery.

ISBN 9781450326568.

10.1145/2555243.2555258.

URL https://doi.org/10.1145/2555243.2555258.

  • Bisk et al. (2020)

Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi.

PIQA：关于自然语言中物理常识的推理。

In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020.

10.1609/aaai.v34i05.6239.

URL https://doi.org/10.1609/aaai.v34i05.6239.

  • Chen et al. (2021)

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba.

评估在代码上训练的大语言模型。

CoRR, abs/2107.03374, 2021.

URL https://arxiv.org/abs/2107.03374.

  • Clark et al. (2018)

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord.

认为你已解决问答？试试 ARC，这个 AI2 推理挑战。

CoRR, abs/1803.05457, 2018.

URL http://arxiv.org/abs/1803.05457.

  • Cobbe et al. (2021)

K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al.

训练验证器以解决数学应用题。

arXiv preprint arXiv:2110.14168, 2021.

  • Cui et al. (2019)

Y. Cui, T. Liu, W. Che, L. Xiao, Z. Chen, W. Ma, S. Wang, and G. Hu.

一个用于中文机器阅读理解的指代抽取数据集。

In K. Inui, J. Jiang, V. Ng, and X. Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5883–5889, Hong Kong, China, Nov. 2019. Association for Computational Linguistics.

10.18653/v1/D19-1600.

URL https://aclanthology.org/D19-1600.

  • Dai et al. (2024)

D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y. K. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang.

Deepseekmoe：面向混合专家语言模型中终极专家专门化的探索。

CoRR, abs/2401.06066, 2024.

URL https://doi.org/10.48550/arXiv.2401.06066.

  • DeepSeek-AI (2024a)

DeepSeek-AI.

Deepseek-coder-v2：打破闭源模型在代码智能上的壁垒。

CoRR, abs/2406.11931, 2024a.

URL https://doi.org/10.48550/arXiv.2406.11931.

  • DeepSeek-AI (2024b)

DeepSeek-AI.

Deepseek LLM：以长期主义扩展开源语言模型。

CoRR, abs/2401.02954, 2024b.

URL https://doi.org/10.48550/arXiv.2401.02954.

  • DeepSeek-AI (2024c)

DeepSeek-AI.

Deepseek-v2：一个强大、经济且高效的混合专家语言模型。

CoRR, abs/2405.04434, 2024c.

URL https://doi.org/10.48550/arXiv.2405.04434.

  • Dettmers et al. (2022)

T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer.

Gpt3.int8()：面向 Transformer 的大规模 8 位矩阵乘法。

Advances in Neural Information Processing Systems, 35:30318–30332, 2022.

  • Ding et al. (2024)

H. Ding, Z. Wang, G. Paolini, V. Kumar, A. Deoras, D. Roth, and S. Soatto.

更少的截断改进语言建模。

arXiv preprint arXiv:2404.10830, 2024.

  • Dua et al. (2019)

D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner.

DROP：一个需要对段落进行离散推理的阅读理解基准。

In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368–2378. Association for Computational Linguistics, 2019.

10.18653/V1/N19-1246.

URL https://doi.org/10.18653/v1/n19-1246.

  • Dubois et al. (2024)

Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto.

长度控制的 alpacaeval：一种消除自动评估器偏差的简单方法。

arXiv preprint arXiv:2404.04475, 2024.

  • Fedus et al. (2021)

W. Fedus, B. Zoph, and N. Shazeer.

Switch transformers：用简单高效的稀疏性扩展到万亿参数模型。

CoRR, abs/2101.03961, 2021.

URL https://arxiv.org/abs/2101.03961.

  • Fishman et al. (2024)

M. Fishman, B. Chmiel, R. Banner, and D. Soudry.

将 FP8 训练扩展到万亿词元 LLM。

arXiv preprint arXiv:2409.12517, 2024.

  • Frantar et al. (2022)

E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh.

Gptq：面向生成式预训练 Transformer 的精确训练后量化。

arXiv preprint arXiv:2210.17323, 2022.

  • Gao et al. (2020)

L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al.

The Pile：一个用于语言建模的 800GB 多样化文本数据集。

arXiv preprint arXiv:2101.00027, 2020.

  • Gema et al. (2024)

A. P. Gema, J. O. J. Leang, G. Hong, A. Devoto, A. C. M. Mancino, R. Saxena, X. He, Y. Zhao, X. Du, M. R. G. Madani, C. Barale, R. McHardy, J. Harris, J. Kaddour, E. van Krieken, and P. Minervini.

我们是否已经完成了 MMLU？

CoRR, abs/2406.04127, 2024.

URL https://doi.org/10.48550/arXiv.2406.04127.

  • Gloeckle et al. (2024)

F. Gloeckle, B. Y. Idrissi, B. Rozière, D. Lopez-Paz, and G. Synnaeve.

通过多词元预测获得更好更快的大语言模型。

In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.

URL https://openreview.net/forum?id=pEWAcejiU2.

  • Google (2024)

Google.

我们的下一代模型：Gemini 1.5, 2024.

URL https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024.

  • Graham et al. (2016)

R. L. Graham, D. Bureddy, P. Lui, H. Rosenstock, G. Shainer, G. Bloch, D. Goldenerg, M. Dubman, S. Kotchubievsky, V. Koushnir, et al.

可扩展的分层聚合协议（SHArP）：一种用于高效数据归约的硬件架构。

In 2016 First International Workshop on Communication Optimizations in HPC (COMHPC), pages 1–10. IEEE, 2016.

  • Gu et al. (2024)

A. Gu, B. Rozière, H. Leather, A. Solar-Lezama, G. Synnaeve, and S. I. Wang.

Cruxeval：一个用于代码推理、理解与执行的基准, 2024.

  • Guo et al. (2024)

D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, and W. Liang.

Deepseek-coder：当大语言模型遇上编程——代码智能的崛起。

CoRR, abs/2401.14196, 2024.

URL https://doi.org/10.48550/arXiv.2401.14196.

  • Harlap et al. (2018)

A. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. Devanur, G. Ganger, and P. Gibbons.

Pipedream：快速高效的流水线并行 DNN 训练, 2018.

URL https://arxiv.org/abs/1806.03377.

  • (32)

B. He, L. Noci, D. Paliotta, I. Schlag, and T. Hofmann.

理解并最小化 Transformer 训练中的离群特征。

In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

  • He et al. (2024)

Y. He, S. Li, J. Liu, Y. Tan, W. Wang, H. Huang, X. Bu, H. Guo, C. Hu, B. Zheng, et al.

中文 simpleqa：面向大语言模型的中文事实性评估。

arXiv preprint arXiv:2411.07140, 2024.

  • Hendrycks et al. (2020)

D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt.

测量大规模多任务语言理解。

arXiv preprint arXiv:2009.03300, 2020.

  • Hendrycks et al. (2021)

D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.

用 MATH 数据集衡量数学问题求解能力。

arXiv preprint arXiv:2103.03874, 2021.

  • Huang et al. (2023)

Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al.

C-Eval：一个面向基础模型的多层次多学科中文评估套件。

arXiv preprint arXiv:2305.08322, 2023.

  • Jain et al. (2024)

N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica.

Livecodebench：面向代码的大语言模型的整体且无污染评估。

CoRR, abs/2403.07974, 2024.

URL https://doi.org/10.48550/arXiv.2403.07974.

  • Jiang et al. (2023)

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al.

Mistral 7b.

arXiv preprint arXiv:2310.06825, 2023.

  • Joshi et al. (2017)

M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer.

TriviaQA：一个用于阅读理解的大规模远程监督挑战数据集。

In R. Barzilay and M.-Y. Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada, July 2017. Association for Computational Linguistics.

10.18653/v1/P17-1147.

URL https://aclanthology.org/P17-1147.

  • Kalamkar et al. (2019)

D. Kalamkar, D. Mudigere, N. Mellempudi, D. Das, K. Banerjee, S. Avancha, D. T. Vooturi, N. Jammalamadaka, J. Huang, H. Yuen, et al.

一项关于 bfloat16 用于深度学习训练的研究。

arXiv preprint arXiv:1905.12322, 2019.

  • Krishna et al. (2024)

S. Krishna, K. Krishna, A. Mohananey, S. Schwarcz, A. Stambler, S. Upadhyay, and M. Faruqui.

事实、获取与推理：检索增强生成的统一评估。

CoRR, abs/2409.12941, 2024.

10.48550/ARXIV.2409.12941.

URL https://doi.org/10.48550/arXiv.2409.12941.

  • Kwiatkowski et al. (2019)

T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. P. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov.

Natural questions：一个用于问答研究的基准。

Trans. Assoc. Comput. Linguistics, 7:452–466, 2019.

10.1162/tacl_a_00276.

URL https://doi.org/10.1162/tacl_a_00276.

  • Lai et al. (2017)

G. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy.

RACE：来自考试的大规模阅读理解数据集。

In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017.

10.18653/V1/D17-1082.

URL https://doi.org/10.18653/v1/d17-1082.

  • Lambert et al. (2024)

N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, et al.

Rewardbench：评估用于语言建模的奖励模型。

arXiv preprint arXiv:2403.13787, 2024.

  • Lepikhin et al. (2021)

D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.

Gshard：用条件计算和自动分片扩展巨型模型。

In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.

URL https://openreview.net/forum?id=qrwe7XHTmYb.

  • Leviathan et al. (2023)

Y. Leviathan, M. Kalman, and Y. Matias.

通过推测解码实现来自 Transformer 的快速推理。

In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR, 2023.

URL https://proceedings.mlr.press/v202/leviathan23a.html.

  • Li et al. (2023)

H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin.

CMMLU：测量中文中的大规模多任务语言理解。

arXiv preprint arXiv:2306.09212, 2023.

  • Li and Hoefler (2021)

S. Li and T. Hoefler.

Chimera：用双向流水线高效训练大规模神经网络。

In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’21, page 1–14. ACM, Nov. 2021.

10.1145/3458817.3476145.

URL http://dx.doi.org/10.1145/3458817.3476145.

  • Li et al. (2024a)

T. Li, W.-L. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica.

从众包数据到高质量基准：Arena-hard 与 benchbuilder 流水线。

arXiv preprint arXiv:2406.11939, 2024a.

  • Li et al. (2021)

W. Li, F. Qi, M. Sun, X. Yi, and J. Zhang.

Ccpm：一个中文古典诗歌匹配数据集, 2021.

  • Li et al. (2024b)

Y. Li, F. Wei, C. Zhang, and H. Zhang.

EAGLE：推测采样需要重新思考特征不确定性。

In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024b.

URL https://openreview.net/forum?id=1NdN7eXyb4.

  • Lin (2024)

B. Y. Lin.

ZeroEval：一个用于评估语言模型的统一框架, July 2024.

URL https://github.com/WildEval/ZeroEval.

  • Loshchilov and Hutter (2017)

I. Loshchilov and F. Hutter.

解耦权重衰减正则化。

arXiv preprint arXiv:1711.05101, 2017.

  • Lundberg (2023)

S. Lundberg.

提示设计的艺术：提示边界与词元修复, 2023.

URL https://towardsdatascience.com/the-art-of-prompt-design-prompt-boundaries-and-token-healing-3b2448b0be38.

  • Luo et al. (2024)

Y. Luo, Z. Zhang, R. Wu, H. Liu, Y. Jin, K. Zheng, M. Wang, Z. He, G. Hu, L. Chen, et al.

面向深度学习的 Ascend HiFloat8 格式。

arXiv preprint arXiv:2409.16626, 2024.

  • MAA (2024)

MAA.

美国数学邀请赛 - AIME。

In American Invitational Mathematics Examination - AIME 2024, February 2024.

URL https://maa.org/math-competitions/american-invitational-mathematics-examination-aime.

  • Micikevicius et al. (2022)

P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Heinecke, P. Judd, J. Kamalu, et al.

面向深度学习的 FP8 格式。

arXiv preprint arXiv:2209.05433, 2022.

  • Mistral (2024)

Mistral.

更便宜、更好、更快、更强：继续推动 AI 的前沿并使其惠及所有人, 2024.

URL https://mistral.ai/news/mixtral-8x22b.

  • Narang et al. (2017)

S. Narang, G. Diamos, E. Elsen, P. Micikevicius, J. Alben, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, et al.

混合精度训练。

In Int. Conf. on Learning Representation, 2017.

  • Noune et al. (2022)

B. Noune, P. Jones, D. Justus, D. Masters, and C. Luschi.

用于深度神经网络的 8 位数值格式。

arXiv preprint arXiv:2206.02915, 2022.

  • NVIDIA (2022)

NVIDIA.

利用 NVIDIA Magnum IO NVSHMEM 和 GPUDirect Async 提升 HPC 系统的网络性能。

https://developer.nvidia.com/blog/improving-network-performance-of-hpc-systems-using-nvidia-magnum-io-nvshmem-and-gpudirect-async, 2022.

  • NVIDIA (2024a)

NVIDIA.

Blackwell 架构。

https://www.nvidia.com/en-us/data-center/technologies/blackwell-architecture/, 2024a.

  • NVIDIA (2024b)

NVIDIA.

TransformerEngine, 2024b.

URL https://github.com/NVIDIA/TransformerEngine.

Accessed: 2024-11-19.

  • OpenAI (2024a)

OpenAI.

你好 GPT-4o, 2024a.

URL https://openai.com/index/hello-gpt-4o/.

  • OpenAI (2024b)

OpenAI.

多语言大规模多任务语言理解（mmmlu）, 2024b.

URL https://huggingface.co/datasets/openai/MMMLU.

  • OpenAI (2024c)

OpenAI.

推出 SimpleQA, 2024c.

URL https://openai.com/index/introducing-simpleqa/.

  • OpenAI (2024d)

OpenAI.

推出 SWE-bench verified，我们将发布一个经过人工验证的 swe-bench 子集，它更, 2024d.

URL https://openai.com/index/introducing-swe-bench-verified/.

  • Peng et al. (2023a)

B. Peng, J. Quesnelle, H. Fan, and E. Shippole.

Yarn：大语言模型高效的上下文窗口扩展。

arXiv preprint arXiv:2309.00071, 2023a.

  • Peng et al. (2023b)

H. Peng, K. Wu, Y. Wei, G. Zhao, Y. Yang, Z. Liu, Y. Xiong, Z. Yang, B. Ni, J. Hu, et al.

FP8-LM：训练 FP8 大语言模型。

arXiv preprint arXiv:2310.18313, 2023b.

  • Qi et al. (2023a)

P. Qi, X. Wan, G. Huang, and M. Lin.

零气泡流水线并行。

arXiv preprint arXiv:2401.10241, 2023a.

  • Qi et al. (2023b)

P. Qi, X. Wan, G. Huang, and M. Lin.

零气泡流水线并行, 2023b.

URL https://arxiv.org/abs/2401.10241.

  • Qwen (2023)

Qwen.

Qwen 技术报告。

arXiv preprint arXiv:2309.16609, 2023.

  • Qwen (2024a)

Qwen.

推出 Qwen1.5, 2024a.

URL https://qwenlm.github.io/blog/qwen1.5.

  • Qwen (2024b)

Qwen.

Qwen2.5：一个基础模型家族, 2024b.

URL https://qwenlm.github.io/blog/qwen2.5.

  • Rajbhandari et al. (2020)

S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He.

Zero：面向训练万亿参数模型的内存优化。

In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020.

  • Rein et al. (2023)

D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman.

GPQA：一个研究生级别的防搜索问答基准。

arXiv preprint arXiv:2311.12022, 2023.

  • Rouhani et al. (2023a)

B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolf, et al.

用于深度学习的微缩放数据格式。

arXiv preprint arXiv:2310.10537, 2023a.

  • Rouhani et al. (2023b)

B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolf, et al.

用于深度学习的微缩放数据格式。

arXiv preprint arXiv:2310.10537, 2023b.

  • Sakaguchi et al. (2019)

K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi.

Winogrande：大规模对抗性 Winograd 模式挑战, 2019.

  • Shao et al. (2024)

Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo.

Deepseekmath：在开放语言模型中突破数学推理的极限。

arXiv preprint arXiv:2402.03300, 2024.

  • Shazeer et al. (2017)

N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean.

极其庞大的神经网络：稀疏门控的混合专家层。

In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017.

URL https://openreview.net/forum?id=B1ckMDqlg.

  • Shi et al. (2023)

F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, D. Das, and J. Wei.

语言模型是多语言思维链推理者。

In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.

URL https://openreview.net/forum?id=fR3wGCk-IXp.

  • Shibata et al. (1999)

Y. Shibata, T. Kida, S. Fukamachi, M. Takeda, A. Shinohara, T. Shinohara, and S. Arikawa.

字节对编码：一种加速模式匹配的文本压缩方案。

1999.

  • Su et al. (2024)

J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu.

Roformer：增强型旋转位置嵌入 Transformer。

Neurocomputing, 568:127063, 2024.

  • Sun et al. (2019a)

K. Sun, D. Yu, D. Yu, and C. Cardie.

研究用于挑战性中文机器阅读理解的先验知识, 2019a.

  • Sun et al. (2024)

M. Sun, X. Chen, J. Z. Kolter, and Z. Liu.

大语言模型中的大规模激活。

arXiv preprint arXiv:2402.17762, 2024.

  • Sun et al. (2019b)

X. Sun, J. Choi, C.-Y. Chen, N. Wang, S. Venkataramani, V. V. Srinivasan, X. Cui, W. Zhang, and K. Gopalakrishnan.

混合 8 位浮点（HFP8）深度神经网络的训练与推理。

Advances in neural information processing systems, 32, 2019b.

  • Suzgun et al. (2022)

M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al.

具有挑战性的 big-bench 任务以及思维链能否解决它们。

arXiv preprint arXiv:2210.09261, 2022.

  • Thakkar et al. (2023)

V. Thakkar, P. Ramani, C. Cecka, A. Shivam, H. Lu, E. Yan, J. Kosaian, M. Hoemmen, H. Wu, A. Kerr, M. Nicely, D. Merrill, D. Blasig, F. Qiao, P. Majcher, P. Springer, M. Hohnerbach, J. Wang, and M. Gupta.

CUTLASS, Jan. 2023.

URL https://github.com/NVIDIA/cutlass.

  • Touvron et al. (2023a)

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.

LLaMA：开放且高效的基础语言模型。

arXiv preprint arXiv:2302.13971, 2023a.

  • Touvron et al. (2023b)

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom.

Llama 2：开放的基础模型与微调聊天模型。

CoRR, abs/2307.09288, 2023b.

10.48550/arXiv.2307.09288.

URL https://doi.org/10.48550/arXiv.2307.09288.

  • Vaswani et al. (2017)

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.

Attention is all you need（注意力即你所需要的一切）。

Advances in neural information processing systems, 30, 2017.

  • Wang et al. (2024a)

L. Wang, H. Gao, C. Zhao, X. Sun, and D. Dai.

面向混合专家的无辅助损失负载均衡策略。

CoRR, abs/2408.15664, 2024a.

URL https://doi.org/10.48550/arXiv.2408.15664.

  • Wang et al. (2024b)

Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen.

Mmlu-pro：一个更稳健且更具挑战性的多任务语言理解基准。

CoRR, abs/2406.01574, 2024b.

URL https://doi.org/10.48550/arXiv.2406.01574.

  • Wei et al. (2023)

T. Wei, J. Luan, W. Liu, S. Dong, and B. Wang.

Cmath：你的语言模型能通过中国小学数学测试吗？, 2023.

  • Wortsman et al. (2023)

M. Wortsman, T. Dettmers, L. Zettlemoyer, A. Morcos, A. Farhadi, and L. Schmidt.

面向大规模视觉-语言模型的稳定且低精度的训练。

Advances in Neural Information Processing Systems, 36:10271–10298, 2023.

  • Xi et al. (2023)

H. Xi, C. Li, J. Chen, and J. Zhu.

用 4 位整数训练 Transformer。

Advances in Neural Information Processing Systems, 36:49146–49168, 2023.

  • Xia et al. (2024)

C. S. Xia, Y. Deng, S. Dunn, and L. Zhang.

Agentless：揭开基于 LLM 的软件工程智能体的神秘面纱。

arXiv preprint, 2024.

  • Xia et al. (2023)

H. Xia, T. Ge, P. Wang, S. Chen, F. Wei, and Z. Sui.

推测解码：利用推测执行加速 seq2seq 生成。

In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 3909–3925. Association for Computational Linguistics, 2023.

URL https://doi.org/10.18653/v1/2023.findings-emnlp.257.

  • Xiao et al. (2023)

G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han.

Smoothquant：面向大语言模型的精确且高效的训练后量化。

In International Conference on Machine Learning, pages 38087–38099. PMLR, 2023.

  • Xu et al. (2020)

L. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan.

CLUE：一个中文语言理解评估基准。

In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020.

10.18653/V1/2020.COLING-MAIN.419.

URL https://doi.org/10.18653/v1/2020.coling-main.419.

  • Zellers et al. (2019)

R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi.

HellaSwag：机器真的能完成你的句子吗？

In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019.

10.18653/v1/p19-1472.

URL https://doi.org/10.18653/v1/p19-1472.

  • Zhong et al. (2023)

W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan.

AGIEval：一个以人为中心的评估基础模型的基准。

CoRR, abs/2304.06364, 2023.

10.48550/arXiv.2304.06364.

URL https://doi.org/10.48550/arXiv.2304.06364.

  • Zhou et al. (2023)

J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou.

面向大语言模型的指令遵循评估。

arXiv preprint arXiv:2311.07911, 2023.