论文信息与摘要（原文）

原文标题：DeepSeek-V3 Technical Report arXiv：2412.19437（v1）　方向：第三批 大模型基座：开源前沿再跃迁

以下为 arXiv 官方 HTML 全文，一字不落保留原文结构、图表与参考文献，仅供学习研究参考，版权归原作者所有。

\reportnumber 001

DeepSeek-AI

research@deepseek.com

Abstract

We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks. The model checkpoints are available at https://github.com/deepseek-ai/DeepSeek-V3.

图

Figure 1: Benchmark performance of DeepSeek-V3 and its counterparts.

Contents

  • 1 Introduction

  • 2 Architecture

  • 2.1 Basic Architecture

  • 2.1.1 Multi-Head Latent Attention

  • 2.1.2 DeepSeekMoE with Auxiliary-Loss-Free Load Balancing

  • 2.2 Multi-Token Prediction

  • 3 Infrastructures

  • 3.1 Compute Clusters

  • 3.2 Training Framework

  • 3.2.1 DualPipe and Computation-Communication Overlap

  • 3.2.2 Efficient Implementation of Cross-Node All-to-All Communication

  • 3.2.3 Extremely Memory Saving with Minimal Overhead

  • 3.3 FP8 Training

  • 3.3.1 Mixed Precision Framework

  • 3.3.2 Improved Precision from Quantization and Multiplication

  • 3.3.3 Low-Precision Storage and Communication

  • 3.4 Inference and Deployment

  • 3.4.1 Prefilling

  • 3.4.2 Decoding

  • 3.5 Suggestions on Hardware Design

  • 3.5.1 Communication Hardware

  • 3.5.2 Compute Hardware

  • 4 Pre-Training

  • 4.1 Data Construction

  • 4.2 Hyper-Parameters

  • 4.3 Long Context Extension

  • 4.4 Evaluations

  • 4.4.1 Evaluation Benchmarks

  • 4.4.2 Evaluation Results

  • 4.5 Discussion

  • 4.5.1 Ablation Studies for Multi-Token Prediction

  • 4.5.2 Ablation Studies for the Auxiliary-Loss-Free Balancing Strategy

  • 4.5.3 Batch-Wise Load Balance VS. Sequence-Wise Load Balance

  • 5 Post-Training

  • 5.1 Supervised Fine-Tuning

  • 5.2 Reinforcement Learning

  • 5.2.1 Reward Model

  • 5.2.2 Group Relative Policy Optimization

  • 5.3 Evaluations

  • 5.3.1 Evaluation Settings

  • 5.3.2 Standard Evaluation

  • 5.3.3 Open-Ended Evaluation

  • 5.3.4 DeepSeek-V3 as a Generative Reward Model

  • 5.4 Discussion

  • 5.4.1 Distillation from DeepSeek-R1

  • 5.4.2 Self-Rewarding

  • 5.4.3 Multi-Token Prediction Evaluation

  • 6 Conclusion, Limitations, and Future Directions

  • References

  • A Contributions and Acknowledgments

  • B Ablation Studies for Low-Precision Training

  • B.1 FP8 v.s. BF16 Training

  • B.2 Discussion About Block-Wise Quantization

  • C Expert Specialization Patterns of the 16B Aux-Loss-Based and Aux-Loss-Free Models