4 Model Design and Acceleration

本文为论文《Wan 2.1：开放先进的视频生成基座模型》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

4.1 Spatio-temporal Variational Autoencoder

Figure 5: Our Wan-VAE Framework. Wan-VAE can compress the spatio-temporal dimension of a video by $4\times 8\times 8$ times. The orange rectangles represent $2\times$ spatio-temporal compression, and the green rectangles represent $2\times$ spatial compression.

Variational Autoencoders (VAEs) (Wu et al., 2024; Blattmann et al., 2023; OpenAI, 2024) play a crucial role in learning compact latent representations from high-dimensional visual data (especially videos), facilitating scalable and efficient training of generative models, e.g., diffusion models. However, designing effective VAEs for video generation tasks faces several challenges. First, videos inherently possess both spatial and temporal dimensions, requiring the VAE to capture complex spatio-temporal dependencies. Second, the inherent high-dimensional nature of video (e.g., multiple frames with high-resolution pixels) increases memory consumption and computational costs, making it difficult to scale VAEs to long video sequences. Third, ensuring temporal causality (i.e., future frames do not influence past frames) is critical for generating realistic and coherent video content, yet this constraint introduces additional architectural complexities.

To address these challenges, we propose a novel 3D causal VAE architecture (Wan-VAE) specifically designed for video generation. We combine multiple strategies (Wu et al., 2024) to improve spatio-temporal compression, reduce memory usage, and ensure temporal causality. These enhancements make Wan-VAE more efficient and scalable and better suited for integration with diffusion-based generative models such as DiT. Next, we introduce the network design, training details, efficient inference, and experimental comparison of our Wan-VAE.

4.1.1 Model Design

To achieve a bidirectional mapping between the high-dimensional pixel space and the low-dimensional latent space, we design a 3D causal VAE as illustrated in Fig. 5. Given an input video $V\in\mathbb{R}^{(1+T)\times H\times W\times 3}$, Wan-VAE compresses its spatio-temporal dimensions to $[1+T/4,H/8,W/8]$ while expanding the number of channels $C$ to 16. Specifically, the first frame is only spatially compressed to better handle the image data, following MagViT-v2 (Yu et al., 2023). In terms of architecture, we replace all GroupNorm layers (Wu & He, 2018) with RMSNorm layers (Zhang & Sennrich, 2019) to preserve temporal causality. This modification enables the use of the feature cache mechanism (refer to Sec. 4.1.3), significantly improving the inference efficiency. Furthermore, we halve the input feature channel in the spatial upsampling layer, resulting in a 33% reduction in memory consumption during inference.

By carefully tuning the number of base channels further, Wan-VAE achieves a compact model size of only 127M parameters. This optimization reduces the encoding time and memory usage, thus benefiting the training of subsequent diffusion transformer models.

4.1.2 Training

We adopt a three-stage approach to train Wan-VAE. First, we construct a 2D image VAE with the same structure and train it on image data. Then, we inflate the well-trained 2D image VAE into a 3D causal Wan-VAE to provide an initial spatial compression prior, which drastically improves the training speed compared to training a video VAE from scratch. At this stage, Wan-VAE is trained on the low-resolution (128$\times$128) and small frame number (5-frame) videos to accelerate convergence speed. The training losses include $L1$ reconstruction loss, KL loss, and LPIPS perceptual loss, which are weighted by coefficients of 3, 3e-6, and 3, respectively. Finally, we fine-tune the model on high-quality videos with different resolutions and frame numbers, integrating the GAN loss (Esser et al., 2021) from a 3D discriminator.

Figure 6: Our feature cache mechanism. (a) and (b) show how we use this mechanism in regular causal convolution and temporal downsampling, respectively.

4.1.3 Efficient Inference

To efficiently support the encoding and decoding of arbitrarily long videos (Yang et al., 2025a; Yao et al., 2021), we implement a feature cache mechanism within the causal convolution module of Wan-VAE.

Specifically, the number of video sequence frames follows the $1+T$ input format, so we divide the video into $1+T/4$ chunks, which is consistent with the number of latent features. When processing input video sequences, the model employs a chunk-wise strategy where each encoding and decoding operation handles only the video chunk corresponding to a single latent representation. Based on the temporal compression ratio, the number of frames in each processing chunk is limited to at most 4, effectively preventing memory overflow.

To ensure temporal continuity between context chunks, our model strategically maintains frame-level feature caches from preceding chunks. These cached features are systematically integrated into the causal convolution computations of subsequent chunks. The specific process is illustrated in Fig. 6, which demonstrates two typical scenarios in our feature cache mechanism. In Fig. 6 (a), in our default setting, the causal convolution does not change the number of frames, and we need to maintain two cached features (convolution kernel size is 3) from historical frames. For the initial chunk, we apply zero padding with two dummy frames to initialize the cache buffer. Subsequent chunks systematically reuse the last two frames from the preceding chunk as cached features while discarding obsolete historical data. Fig. 6 (b) presents the scenario involving 2$\times$ temporal downsampling (stride is equal to 2). Here, the scenario necessitates different cache management: we implement single-frame cache padding exclusively for non-initial chunks to ensure dimensional consistency. This configuration guarantees that the output sequence length follows the exact downsampling ratio while maintaining causal relationships across the boundaries of the chunk.

This feature cache mechanism not only optimizes memory utilization but also preserves feature coherence across chunk boundaries, thereby supporting stable inference for infinite-length videos.

4.1.4 Evaluation

Figure 7: Comparison of video reconstruction performance at $720\times 720$ resolution and 25 frames.

Quantitative results. This study conducts a comprehensive evaluation of the performance of various state-of-the-art (SOTA) video VAE models (Kong et al., 2024; Yang et al., 2025b) by analyzing their Peak Signal-to-Noise Ratio (PSNR) and processing efficiency (frame per unit latency (second)). Notably, for fair comparison, most models (Kong et al., 2024; Yang et al., 2025b) adopt the same compression rate and latent feature dimension as our approach, i.e., a compression rate of $4\times 8\times 8$ with a latent dimension of 16. Open Sora Plan (Lab & etc., 2024) has the same compression ratio as ours, but the latent dimension is 4. SVD (Blattmann et al., 2023) employs a compression rate of $1\times 8\times 8$ with a latent dimension of 4, Step Video (Ma et al., 2025) employs a compression rate of $8\times 16\times 16$ with a latent dimension of 64, and Mochi (GenmoTeam, 2024) uses a compression rate of $6\times 8\times 8$ with a latent dimension of 12. The evaluation is performed on 200 videos. These videos consist of 25 frames, and the resolution is set to $720\times 720$. In Fig. 7, the size of the circles is positively correlated with the number of model parameters.

Experimental results indicate that Wan-VAE exhibits highly competitive performance across both metrics, showcasing a compelling dual advantage of superior video quality and high processing efficiency. It is worth noting that under the same hardware environment, our VAE’s reconstruction speed is 2.5 times faster than the existing SOTA method (i.e., HunYuan Video). This speed advantage will be further demonstrated at higher resolutions due to the small size design of our VAE model and the feature cache mechanism.

This advancement offers an efficient solution for video reconstruction tasks and video generation training. These findings not only validate the effectiveness of the proposed model design but also provide valuable insights for the future development of VAE technologies.

Figure 8: Visualization results of video reconstruction across different scenarios, including texture (first row), face (second row), text (third row), and high-motion (fourth row). The above videos have rich details but are not used in training.

Qualitative results. To validate the video reconstruction performance of our method across diverse scenarios, Fig. 8 presents visual results for texture, face, text, and high-motion scenes. Compared to existing VAE models, Wan-VAE demonstrates superior performance in these contexts. In the texture scene shown in the first row, our method is capable of capturing details more accurately, such as the texture and direction of hair. In the facial scene depicted in the second row, Wan-VAE preserves facial features while reducing blurring and distortion around the lips. In the text scene illustrated in the third row, our method successfully restores textual content with clarity, mitigating issues of character distortion and loss. In the high-motion scene presented in the fourth row, Wan-VAE maintains the motion sharpness of video frames. These results indicate that our method offers significant advantages in handling a variety of complex scenarios.

Figure 9: Architecture of the Wan.

4.2 Model Training

In this section, we will provide a detailed introduction to the architecture design of the foundational video model, specifically for the text-to-video task, along with the details of the pre-training and post-training phases.

In Fig. 9, we present the architecture of Wan, which is designed based on the current mainstream DiT (Peebles & Xie, 2023) architecture, which typically consists of three primary components: Wan-VAE, diffusion transformer, and text encoder. For a given video $V\in\mathbb{R}^{(1+T)\times H\times W\times 3}$, the encoder of Wan-VAE encodes it from pixel space into the latent space $x\in\mathbb{R}^{1+T/4\times H/8\times W/8}$, which is then fed into the subsequent diffusion transformer structure.

4.2.1 Video Diffusion Transformer

Figure 10: Transformer block of Wan.

Diffusion transformer. The diffusion transformer mainly consists of three components: a patchifying module, transformer blocks, and an unpatchifying module. Within each block, we focus on effectively modeling spatio-temporal contextual relationships and embedding text conditions alongside time steps. In the patchifying module, we use a 3D convolution with a kernel size of $(1,2,2)$ and apply a flattening operation to convert $x$ into a sequence of features with the shape of $(B,L,D)$, where $B$ denotes the batch size, $L=(1+T/4)\times H/16\times W/16$ represents the sequence length, and $D$ indicates the latent dimension. Specifically, as illustrated in Fig. 10, we employ the cross-attention mechanism to embed the input text conditions, which can ensure the model’s ability to follow instructions even under long-context modeling. Additionally, we employ an MLP with a Linear layer and a SiLU (Elfwing et al., 2018) layer to process the input time embeddings and predict six modulation parameters individually. This MLP is shared across all transformer blocks, with each block learning a distinct set of biases. Through extensive experiments, we have demonstrated that this design can reduce the parameter count by approximately $25\%$ and reveal a significant performance improvement with this approach at the same parameter scale.

Text encoder. Wan’s architecture uses the umT5 (Chung et al., 2023) to encode input text. Through extensive experiments, we find that umT5 has several advantages in our framework: 1) It possesses strong multilingual encoding capabilities, allowing it to understand both Chinese and English effectively, as well as input visual text; 2) Under the same conditions, we find that umT5 outperforms other unidirectional attention mechanism LLMs in composition; 3) It exhibits superior convergence, with umT5 achieving faster convergence at the same parameter scale. Based on these findings, we ultimately chose umT5 as our text embedder.

4.2.2 Pre-training

We leverage the flow matching framework (Lipman et al., 2022; Esser et al., 2024) to model a unified denoising diffusion process across both image and video domains. We first conduct pre-training on low-resolution images, followed by multi-stage joint optimization of images and videos. The image-video joint training evolves using progressively upscaled data resolutions and extended temporal durations throughout the stages.

Training objective. Flow matching provides a theoretically grounded framework for learning continuous-time generative processes in diffusion models. It circumvents iterative velocity prediction, enabling stable training via ordinary differential equations (ODEs) while maintaining equivalence to maximum likelihood objectives. During training, given an image or video latent $x_{1}$, a random noise $x_{0}\sim\mathcal{N}(0,I)$, and a timestep $t\in[0,1]$ sampled from a logit-normal distribution, an intermediate latent $x_{t}$ is obtained as the training input. Following Rectified Flows (RFs) (Esser et al., 2024), $x_{t}$ is defined as a linear interpolation between $x_{0}$ and $x_{1}$, i.e.,

| | $x_{t}=tx_{1}+(1-t)x_{0}.$ | | (1) |

The groud truth velocity $v_{t}$ is

| | $v_{t}=\frac{dx_{t}}{dt}=x_{1}-x_{0}.$ | | (2) |

The model is trained to predict the velocity, thus, the loss function can be formulated as the mean squared error (MSE) between the model output and $v_{t}$,

| | $\mathcal{L}=\mathbb{E}{x{0},x_{1},c_{txt},t}||u(x_{t},c_{txt},t;\theta)-v_{t}||^{2},$ | | (3) |

where $c_{txt}$ is the umT5 text embedding sequence of 512 tokens long, $\theta$ is the model weights, and $u(x_{t},c_{txt},t;\theta)$ denotes the output velocity predicted by the model.

Image pre-training. Our experimental analysis reveals two critical challenges in direct joint training with high-resolution images and long-duration video sequences: (1) Extended sequence lengths (typically $81$ frames for $1280\times 720$ video) substantially reduce training throughput. Under fixed GPU-hour budgets, this results in insufficient data throughput, thereby impeding model convergence rates. (2) Excessive GPU memory consumption forces suboptimal batch sizes, inducing training instability caused by spikes in gradient variance. To mitigate these issues, we initialize the 14B model training through low-resolution ($256\penalty\ px$) text-to-image pre-training, enforcing cross-modal semantic-textual alignment and geometric structure fidelity before progressively introducing high-res video modalities.

Image-video joint training. Following large-scale $256\penalty\ px$ text-to-image pre-training, we implement staged joint-training with image and video data through a resolution-progressive curriculum. The training protocol comprises three distinct stages differentiated by spatial resolution areas: (1) In the first stage, we conduct joint training with $256\penalty\ px$ resolution images and 5-second video clips ($192\penalty\ px$ resolution, $16$ fps). (2) In the second stage, we initiate spatial resolution scaling by upgrading both image and video resolutions to $480px$ while maintaining a fixed 5-second video duration. (3) In the final phase, we escalate spatial resolution to $720px$ for both images and 5-second video clips.

Pre-training configurations. We employ efficient training at bf16-mixed precision combined with the AdamW (Loshchilov & Hutter, 2017; Kingma & Ba, 2014) optimizer with a weight decay coefficient of $1e^{-3}$. The initial learning rate is set to $1e^{-4}$, and dynamically reduced triggered by plateaus of the FID and CLIP Score metrics.

4.2.3 Post-training

In the post-training stage, we maintain the same model architecture and optimizer configuration from the pre-training stage, initializing the network with the pre-trained checkpoint. We conduct joint training at resolutions of $480px$ and $720px$ using the post-training video dataset detailed in Sec. 3.2.

4.3 Model Scaling and Training Efficiency

4.3.1 Workload Analysis

In Wan, the computational costs associated with the text encoder and VAE encoder are lower than those of the DiT model, which accounts for more than 85% of the overall computation during training. For the DiT model, the main computational cost is given by the expression $L(\alpha bsh^{2}+\beta bs^{2}h)$, where $L$ denotes the number of DiT layers, $b$ is the micro batch size, $s$ indicates the sequence length, and $h$ denotes the hidden dimension size. Here, $\alpha$ corresponds to the cost of the linear layers and $\beta$ to that of the attention layer, for the non-causal attention used in Wan, $\beta$ is 4 for the forward pass and 8 for the backward pass. Unlike general LLM models, the sequence length $s$ processed by Wan often reaches hundreds of thousands or more. Since the computational cost of attention increases quadratically with the number of tokens, while the cost associated with the linear layers increases only linearly, the attention mechanism gradually becomes the training bottleneck. In scenarios where the sequence length reaches 1 million, the computation time for attention can account for up to 95% of the end-to-end training time.

During the training process, only the DiT model is optimized while the text encoder and the VAE encoder remain frozen. Therefore, GPU memory usage is concentrated on training the DiT. The GPU memory usage for the DiT can be expressed as $\gamma Lbsh$, where $\gamma$ depends on the implementation of the DiT layers, L denotes the number of DiT layers. Because video tokens, input prompts, and timesteps are taken as model input, the value of $\gamma$ is generally larger than in ordinary LLM models. For example, while standard LLMs might have a $\gamma$ of 34 (Korthikanti et al., ), the DiT model’s $\gamma$ can exceed 60. Consequently, when the sequence length reaches 1 million tokens and the micro-batch size is 1, the total GPU memory usage for activations in a 14B DiT model can exceed 8 TB.

In summary, the primary computational cost in Wan arises from the attention mechanism. While the computational cost increases quadratically with the sequence length $s$, the GPU memory usage scales only linearly with $s$. This provides a valuable reference for guiding subsequent optimization efforts.

4.3.2 Parallelism Strategy

The Wan model mainly consists of three modules: VAE, text encoder, and DiT. Following the cache mechanism optimization outlined in Sec. 4.1.3, the VAE module exhibits minimal GPU memory usage and can seamlessly adopt Data Parallelism (DP). In contrast, the text encoder requires over 20 GB of GPU memory, necessitating the use of model weight sharding to conserve memory for the subsequent DiT module. To address this, we employ a Data Parallel (DP) approach combined with Fully Sharded Data Parallel (FSDP) (Zhao et al., 2023) strategy. The model parameters, gradients, and optimizer states of the DiT will exceed the capacity of a single GPU, with activation storage reaching approximately 8 TB in a scenario involving 1M tokens with a batch size of 1. Moreover, the DiT portion accounts for a significant proportion of the overall computational workload. Thus, we focus on developing efficient distributed strategies specifically for the DiT module.

DiT parallel strategy. In light of the DiT model’s modest size, and considering the optimization of overlapping computation and communication as well as maximizing the size of data parallelism (DP), we have selected FSDP as our parameter sharding strategy.

Regarding activations, the input shape of a DiT block is $[b,s,h]$, where the $b$ dimension corresponds to data parallelism, $s$ represents the sequence length, and $h$ denotes the hidden dimension. Therefore, we need to examine sharding strategies for the $s$ and $h$ dimensions. Sharding along the s dimension is achieved through Context Parallelism (CP), with common methods including Ulysses (Jacobs et al., 2023) and Ring Attention (Liu et al., 2023a). Sharding along the $h$ dimension primarily involves Megatron’s tensor parallelism (TP) (Shoeybi et al., 2019) combined with sequence parallelism (SP) (Korthikanti et al., ), which shards the hidden dimension of the activations by splitting the weights. Since the communication overhead of CP is smaller than that of (TP $+$ SP), we propose using CP to accelerate the batch-level computations while reducing the activation memory footprint on each GPU. We have designed a two-dimensional (2D) CP that combines the characteristics of Ulysses and Ring Attention, similar to USP (Fang & Zhao, 2024). In this design, the outer layer employs Ring Attention, and the inner layer leverages Ulysses. This combination mitigates the slow cross-machine communication inherent in Ulysses and addresses the need for large block sizes after sharding in Ring Attention, thereby maximizing the overlap between outer-layer communication and inner-layer computation. In scenarios with a sequence length of 256K and 16 GPUs across 2 machines, the communication overhead of 2D Context Parallelism is reduced from over $10\%$ with Ulysses to below $1\%$.

Figure 11: Illustration of DiT parallelism. Assuming a total of 128 GPUs, the innermost layer consists of a combination of Ulysses=8 and Ring=2. The outer layer employs FSDP=32, while the outermost layer utilizes DP=4. The global batch size is 8 times that of the micro-batch size.

The FSDP group and CP group intersect within the FSDP group, the DP size equals the FSDP size divided by the CP size. After satisfying memory and single-batch latency requirements, we use DP for the scaling. Fig. 11 illustrates the distributed scheme for the DiT module. For example, in a configuration with 128 GPUs, the CP size is 16, with Ulysses set to 8 and Ring Attention to 2; FSDP is set to 32, corresponding to a batch size of 2b; DP is set to 4, resulting in a global batch size of 8b.

Distributed strategy switching for different modules. During training, different modules utilize the same resources. Specifically, we apply DP to the VAE and Text Encoder, while the DiT module employs a combination of DP and CP. Therefore, when the outputs of the VAE and Text Encoder are fed into the DiT module during training, it is necessary to switch distributed strategies to avoid resource wastage. Specifically, CP requires that devices within the CP group read the same batch of data. Yet, if devices in the VAE and Text Encoder sections also read the same data, this leads to redundant computations. To eliminate this redundancy, devices within the CP group initially read different data. Then, before CP, we perform a loop traversal of size equal to the CP size, sequentially broadcasting the data read by different devices within the CP group to ensure that the inputs within CP are identical. In this way, the time proportion of VAE and Text Encoder within a single model iteration is reduced to $1/CP$, thereby enhancing overall performance.

4.3.3 Memory Optimization

As introduced, the computational cost in Wan grows quadratically with sequence length $s$, whereas GPU memory usage scales linearly with $s$. This means that in long sequence scenarios, computation time can eventually exceed the PCIe transfer time required for offloading activations. Specifically, the PCIe transfer time for offloading the activation of one DiT layer can be overlapped with the computation of just 1 to 3 DiT layers. Compared to gradient checkpointing (GC) strategies (Chen et al., 2016), activation offloading (Rhu et al., 2016) that allows computation overlap provides an effective way to reduce GPU memory usage without sacrificing end-to-end performance. Therefore, we prioritize activation offloading to reduce GPU memory. Since CPU memory is also prone to exhaustion in long sequence scenarios, we combine offloading with GC strategies, prioritizing gradient checkpointing for layers with high GPU memory-to-computation ratios.

4.3.4 Cluster Reliability

By leveraging Alibaba Cloud’s advanced intelligent scheduling, slow machine detection, and robust self-healing capabilities within the training cluster, high stability is ensured. Hardware issues are identified during the job startup phase, ensuring that only healthy nodes are allocated to training tasks. Throughout training, any faulty nodes are promptly isolated and repaired, with tasks automatically restarted and training seamlessly resumed. This efficient orchestration effectively combines reliability with high performance, ensuring overall system stability.

4.4 Inference

The primary objective of inference optimization is to minimize the latency of video generation. Unlike training, the inference process involves multiple sampling steps, typically around 50. Accordingly, we employ techniques such as quantization and distributed computing to reduce the time required for each individual step. Additionally, we leverage the attention similarities between steps to decrease the overall computational load. Furthermore, for classifier-free guidance (CFG) (Ho & Salimans, 2022), we exploit its inherent similarities to further minimize computational requirements.

4.4.1 Parallel Strategy

Figure 12: Scaling inference via multiple GPUs.

As discussed in Sec. 4.3.2, we utilize Context Parallel to reduce the latency of generating a single video when scaling to multiple GPUs. Additionally, for large models like Wan 14B, we adopt model sharding to mitigate GPU memory constraints. Model Sharding Strategy: Given the long sequence lengths during inference, FSDP incurs less communication overhead compared to TP and allows for computation overlap. Therefore, we adopt the FSDP method for model sharding, consistent with our training approach. Context Parallel Strategy: We employ the same 2D Context Parallelism as that in the training stage, with RingAttention serving as the outer loop and Ulysses operating as the inner loop. Equipped with 2D Context Parallel and FSDP parallel strategy, DiT achieves nearly linear speedup on the Wan 14B model, as shown in the Fig. 12.

4.4.2 Diffusion Cache

We conduct a thorough analysis of the Wan model and identify the following characteristics in its inference process:

• •

Attention similarity: Within the same DiT block, attention outputs across different sampling steps exhibit significant similarity.

• •

CFG similarity: In the later stages of sampling, there is a notable similarity between conditional and unconditional DiT outputs.

These findings are also highlighted in recent works, such as DiTFastAttn (Yuan et al., 2024b) and FasterCache (Lv et al., 2024). We leverage these characteristics to implement diffusion caching to reduce the computational workload. In practice, we tailor the diffusion cache to the Wan model to ensure lossless performance. Specifically, we use the validation set to select certain steps for caching:

• •

Attention cache: For the attention component, we perform an attention forward pass every few steps and cache the results, reusing the cached results for the other steps.

• •

CFG cache: For the unconditional part, we perform a DiT forward pass every few steps and reuse the conditional results for the other steps. Additionally, we apply residual compensation methods similar to FasterCache to prevent the degradation of fine details.

After applying diffusion caching to the Wan 14B text-to-video model, we have improved the inference performance by 1.62×.

4.4.3 Quantization

FP8 GEMM. We found that using FP8 precision for the GEMM operations, combined with per-tensor quantization on the weights and per-token quantization on the activations in the sampling steps, incurs minimal performance loss. Thus we apply FP8 quantization to all GEMM operations in the DiT block. The FP8 GEMM delivers twice the performance of BF16 GEMM and achieves a 1.13× speedup in the DiT module.

8-Bit FlashAttention. Although FlashAttention3 (Shah et al., 2025) achieves high performance, its native FP8 implementation suffers from significant quality degradation in video generation, as observed in our experiments. Conversely, SageAttention (Zhang et al., 2024) employs a mixed precision of int8 and fp16 to reduce precision loss, but it does not provide specific adaptations for Hopper GPUs. As of this report’s publication, subsequent optimizations to SageAttention have demonstrated improved Hopper compatibility. Future work will explore the integration of these performance enhancements.

We present our optimizations applied to FA3-FP8 implementation, focusing on both numerical stability and computational efficiency.

Accuracy. To mitigate the numerical errors in attention observed in Wan under FA3-FP8 quantization, we apply two primary techniques:

• •

Mixed 8-Bit optimization. The native implementation of FA3 uses FP8 (E4M3) for all input tensors ($Q$, $K$, $V$). Based on both our analysis of the data distribution and the empirical evidence from the SageAttention, we adopt a hybrid quantization strategy: Int8 for $S=QK^{T}$, FP8 for $O=PV$.

• •

FP32 accumulation for cross-block reduction. The native FP8 WGMMA implementation employs  14-bit accumulators, making it prone to overflow during long-sequence processing. To address this limitation, our approach leverages FP8 WGMMA for intra-block $PV$ reduction while implementing cross-block accumulation through CUDA cores with Float32 registers, a strategy inspired by DeepSeek-V3’s (Liu et al., 2024a) FP8 GEMM methodology.

Performance. To maintain kernel performance under mixed INT8-FP8 quantization while enhancing the accumulation precision of $O=PV$, we employ two techniques:

• •

Fuse Float32 accumulation with intra-warpgroup pipelining. FlashAttention3 employs an intra-warpgroup pipeline that overlaps WGMMA with softmax and scaling. To mitigate the performance penalty introduced by Float32 $PV$ accumulation across blocks, we fuse the Float32 accumulation with $rescale_o$ operation.

• •

Block size tuning. Due to the additional register pressure introduced by Float32 $PV$ accumulation register across blocks, we adjust the block size to reduce register spilling. After optimization, our 8-bit FlashAttention achieves 95% MFU on NVIDIA H20 GPUs.

Consequently, with 8-bit FlashAttention, we boost the inference efficiency by more than 1.27×.

Table 1: Examples of user prompts and their rewritten results. These examples are in Chinese originally and have been translated into English for the purpose of presentation.

4.5 Prompt Alignment

Prompt alignment is designed to boost the effectiveness of model inference by aligning the user’s input prompt with the format and style of captions used during training. We employ two primary strategies in this process. First, we augment each video with a variety of captions, thereby increasing diversity. Second, we rewrite user prompts to align them with the distribution of video captions from the training stage. In the following, we detail these strategies.

To accommodate diverse user inputs, each image or video is paired with multiple captions reflecting various styles and lengths. For example, we use captions of differing lengths (e.g., long, medium, and short) and styles (e.g., formal, informal, and poetic) to create a varied set of sample-caption pairs for our training data. These captions establish different mapping relations between text and video, effectively covering the range of prompts that users might provide.

Despite the diversity of captions, a distribution mismatch often exists between the dataset captions and user input prompts. Typically, users prefer concise prompts consisting of a few simple words, which are significantly shorter than the training captions. This discrepancy can adversely affect the quality of the generated videos. To address this issue, we rewrite prompts by utilizing a large language model (LLM) to refine user prompts. Our primary objective is to align the distribution of these refined prompts with the distribution of the training captions, thereby achieving improved inference performance. The LLM is guided by the following principles when rewriting user prompts:

We instruct the LLM to add details to prompts without altering their original meanings, enhancing the completeness and visual appeal of the generated scenes.

The rewritten prompts should incorporate natural motion attributes, where we add appropriate actions for the subject based on its category to ensure smoother and more fluent motion in the generated videos.

We recommend structuring the rewritten prompts similarly to the post-training captions, beginning with the video style, followed by an abstract of the content, and concluding with a detailed description. This method helps align prompts with the distribution of high-quality video captions.

We evaluate the effectiveness of LLM-assisted prompt rewriting on our benchmark, and several results are presented in Tab. 1. Our findings suggest that LLMs with strong instruction-following capabilities can generate suitable prompts that enhance video generation. To balance speed and performance, we use Qwen2.5-Plus as our rewriting model.

Figure 13: The dimensions covered in Wan-Bench.

Figure 14: Distribution of the number of prompts across various dimensions of Wan-Bench.

4.6 Benchmarks

Existing video generation evaluation methods, such as Fréchet Video Distance (FVD) (Unterthiner et al., 2018) and Fréchet Inception Distance (FID) (Heusel et al., 2017), lack alignment with human perception. We propose an automated, comprehensive, and human-aligned Wan-Bench for evaluating video generation models.

Wan-Bench is composed of three core dimensions: dynamic quality, image quality, and instruction following with 14 fine-grained metrics. We design specific scoring algorithm for each dimension, using traditional detectors for the simple tasks (e.g. , object detection) and multi-modal Large Language Models (MLLMs) for the complex tasks.

Dynamic quality. Dynamic quality aims to evaluate the quality and stability of the model in non-static scenarios.

Large motion generation. To measure the upper limit of motion generation, prompts are designed to encourage significant motions. We calculate the optical flow of the generated video using RAFT (Teed & Deng, 2021) and assess the motion scores by the magnitude of the normalized optical flow.

Human artifacts. Existing evaluation methods lack the ability to detect AI-generated artifacts. Thus, we train a YOLOv3-based model Redmon & Farhadi (2018) on 20,000 AI-generated images annotated by humans to identify the locations of artifacts. The artifact scores take into account the predicted likelihood, the bounding boxes, and the corresponding duration.

Physical plausibility & smoothness. To evaluate physical plausibility, we design prompts to generate physics-related movements (e.g. , ball bouncing, object interactions, water flow) and employ Qwen2-VL (Bai et al., 2023), leveraging its extensive real-world physics knowledge. Through a video question-answer model, we detect violations of physical laws, such as object clipping, unrealistic collisions, or gravity-defying movements. To evaluate smoothness, prompts are crafted to include complex motions, with Qwen2-VL identifying artifacts to assess motion fluidity.

Pixel-level stability. Pixel-level stability is used to determine if noise frequently appears in the video. We employ optical flow to identify static regions and calculate frame-wise differences within those areas. For stable videos, these differences within static regions are relatively minimal.

ID consistency. Three kinds of sub-dimensions are considered: human consistency, animal consistency, and object consistency covering different collections of prompts. Frame-level DINO (Caron et al., 2021) features are extracted to compare the frame-wise similarity as the consistency score.

Image quality. The purpose of the image quality dimension is to evaluate the visual quality and aesthetic perception.

Comprehensive image quality. Measuring the visual quality requires an evaluation of the fidelity and aesthetics. For fidelity, we employ MANIQA (Yang et al., 2022), which effectively detects blur and artifacts, ensuring precise clarity assessment. For aesthetics, we utilize two established models: the LAION-based aesthetic predictor (LAION-AI, 2022) and MUSIQ (Ke et al., 2021). We compute the final image quality score by averaging these three evaluators.

Scene generation quality. We evaluate two key aspects: frame-wise scene consistency and scene-text alignment. For consistency, we measure CLIP similarity (Radford et al., 2021) across consecutive frames to ensure temporal coherence. For alignment, we compute CLIP similarity between frames and their corresponding text to verify semantic accuracy. We compute the quality score by weighted averaging these metrics.

Stylization. We employ Qwen2-VL to evaluate the artistic video generation capability through frame-level question answering.

Instruction following. Instruction following assesses the model’s ability to comprehend and execute textual instructions.

Single object & multiple objects & spatial positions. We employ Qwen2-VL to predict object categories, counts, and spatial relationships within videos. The evaluation score represents the average number of frames that accurately correspond to the given prompts.

Camera control. We evaluate five camera movement styles: panning, lifting, zooming in/out, aerial shots, and tracking shots by 130 tailored prompts. For panning, lifting, and zooming, we analyze optical flow via RAFT to assess movement style. For complex aerial and tracking shots, we employ Qwen2-VL to infer camera movement through video question-answering.

Action instruction following. We categorize motions into human (e.g. , running), animal (e.g. , crawling), and object movements (e.g. , flying). As evaluating these motions requires prior knowledge of their execution, we employ Qwen2-VL by providing it with keyframes. We query the model for alignment of the action, completion of the action, and presence of artifacts to comprehensively evaluate alignment between text and video.

Human feedback guided weighting strategy. We utilize human feedback to assign a comprehensive video quality score, considering that each dimension has a different impact on user preferences rather than simply averaging them. We collect over 5,000 pairwise comparisons of videos generated by different models including Wan. Users evaluate pairs based on a given text, indicating their preference and assigning approximate scores. The dimension preference weighting is derived from the Pearson correlation between model-generated scores and human-rated scores, serving as a weighting factor in the final score calculation.

Table 2: Performance comparison of commercial and open-source models using Wan-Bench.

4.7 Evaluation

4.7.1 Metrics and Results

Baselines and metrics. As of the time of writing, there are numerous comparative competitors in the market, including commercial models such as Kling (Kuaishou, 2024.06), Hailuo (MiniMax, 2024.09), Sora (OpenAI, 2024), Runway Runway (2024.06) and Vidu (ShengShu-AI, 2024.07), as well as open-source models like Mochi (GenmoTeam, 2024), CogVideoX (Yang et al., 2025b) and Hunyuan (Kong et al., 2024). In this section, we compare our model with the commercial and open-source models that have been evaluated through benchmarks and received relatively positive user feedback.

Quantitative results. We collected 1,035 samples for each candidate model using Wan-Bench prompts and conducted fair evaluations by scoring the models with Wan-Bench. The evaluation focused on three key metrics: dynamic quality, image quality, and instruction-following accuracy. We rank the performance of the different generation models by calculating total scores weighted according to human preferences (Sec. 4.6). The results in Table 2 indicate that Wan outperforms existing commercial and open-source models.

Qualitative results. As illustrated in the Fig. 15, Wan effectively generates diverse, high-quality videos directly from textual descriptions. Wan excels at synthesizing dynamic scenes involving large-scale, complex movements, as well as scenarios reflecting physical interactions. Moreover, it adeptly handles various artistic styles and consistently produces cinematic-quality visuals. Additionally, Wan demonstrates robust multilingual text-generation capabilities, integrating Chinese and English texts into animations and giving rise to movie-level textual effects.

Figure 15: Results of the Wan-T2V. Our model excels at generating complex motions, creative transitions, cinematic-quality videos, and accurately produces both Chinese and English text.

Human evaluation. We design over 700 evaluation tasks, which are annotated by more than 20 individuals and assessed by four key dimensions: alignment, image quality, dynamic quality, and overall quality. The results in Table 3 demonstrate that in the T2V task, the Wan 14B model consistently excels across all visual quality dimensions, indicating its superior performance.

Table 3: Win rate gap of T2V.

Wan in public leaderboard. Wan has demonstrated state-of-the-art performance on the VBench leaderboard (Huang et al., 2023), a widely used evaluation approach for video generation task. The VBench employs a multidimensional assessment architecture that decomposes video quality evaluation into 16 distinct, human-aligned dimensions spanning aesthetic quality, motion smoothness, and semantic coherence.

Our analysis focuses on two model variants: the 14B parameter Wan 14B and the Wan 1.3B variant (shown in Table 4). The Wan 14B model achieves a benchmark-leading performance with an aggregate score of 86.22%, including 86.67% in visual quality and 84.44% in semantic consistency, significantly outperforming competitors like OpenAI’s Sora and MiniMax’s Hailuo. The efficient Wan 1.3B variant also remains competitive, scoring 83.96% and surpassing commercial models such as HunyuanVideo (Kong et al., 2024) and Kling 1.0 (Kuaishou, 2024.06), as well as the open-sourced CogVideoX1.5-5B (Yang et al., 2025b).

Table 4: Model performance scores on Vbench.

4.7.2 Ablation Study

We provide ablation studies of key modules within our model design to offer insights into the overall architecture’s contributions. Experiments are conducted on 1.3B version for rapid evaluation.

Ablation on adaptive normalization layers. Given the substantial parameter load of adaptive normalization layers (Perez et al., 2018) (adaLN) in DiT (Peebles & Xie, 2023), we explore whether focusing on parameter volume in adaLN or increasing the depth of the network layers is more effective. To adjust the parameter volume, we follow the setup from PixArt (Chen et al., 2023a) to study whether to share adaLN or not. In the non-shared configuration, each block’s scale and shift parameters are predicted by a block-specific MLP that takes as input the time embedding. Instead, the shared AdaLN configuration (referred to as AdaLN-single in PixArt) computes a single set of global parameters, which involves predicting scale and shift values, in the first block, which are then shared across all blocks. This sharing significantly reduces the number of parameters.

We evaluate four configurations: (i) Full-shared-adaLN-1.3B: This is our original model, where adaLN is fully shared across all 30 attention blocks. (ii) Half-shared-adaLN-1.5B: AdaLN is shared in the first 15 attention blocks while not for the remaining blocks. This results in a 1.5B model with an increase of 0.2B parameters. (iii) Full-shared-adaLN-1.5B (extended): AdaLN is shared across all attention blocks, but the model depth is increased to 35 layers, keeping the 1.5B parameter count. (iv) Non-shared-AdaLN-1.7B: AdaLN is not shared across any blocks, and the model depth remains at 30 layers, resulting in a 1.7B model. We maintain other parameters unchanged and train the models from scratch to perform text-to-image task for 200,000 steps (i.e., the first stage of our text-to-video training), with the global batch size of 1536. For comparison, we measure performance using the L2 loss between generated images and real images in the latent space during the training stage, with smaller training loss indicating better convergence.

Figure 16: Training loss curve with different configurations.

As presented in Fig. 16, the Full-shared-AdaLN-1.3B model, with fewer parameters, results in a slightly higher training loss compared to others. When comparing Half-shared-AdaLN-1.5B and Full-shared-AdaLN-1.5B, which have the same parameter count, Full-shared-AdaLN-1.5B consistently achieves the lowest training loss. This suggests that focusing on model depth rather than the parameters in AdaLN leads to better performance. Additionally, the Non-shared-AdaLN-1.7B model, despite having more parameters, does not outperform the Full-shared-AdaLN-1.5B and further supports this idea. Therefore, we adopt the fully shared AdaLN design to effectively reduce parameter count while preserving performance.

Ablation on text encoder. We select three text encoders that can handle bilingual inputs: umT5 (Chung et al., 2023) (5.3B), Qwen2.5-7B-Instruct (QwenTeam, 2024), and GLM-4-9B (TeamGLM et al., 2024). The T5 series has been a popular choice in video generation models (GenmoTeam, 2024; Yang et al., 2025b), while the latter two encoders excel in language understanding, especially among all LLM models under 10B parameters. For this ablation, we keep other settings unchanged and train a text-to-image task with a global size of 1536. Text embeddings are derived from the second-to-last layers of Qwen2.5-7B-Instruct and GLM-4-9B. We measure the training loss as shown in Fig. 17.

Table 5: FID scores ($\downarrow$) of VAE and VAE-D.

Compared to other powerful LLM-based encoders, umT5 demonstrates superior text embedding performance. Notably, HunyuanVideo (Kong et al., 2024) highlights that decoder-only LLMs use causal attention whereas umT5 adopts bidirectional attention, making it suitable for diffusion models. Following HunyuanVideo’s strategy, we incorporate a bidirectional token refiner as an adapter layer. We apply this setup to both Qwen2.5-7B-Instruct and GLM-4-9B, but the training losses and visualizations still show umT5 performing most favorably. Further, we compare umT5 with a pre-trained Multimodal Large Language Model, i.e., Qwen-VL-7B-Instruct (Bai et al., 2023). As shown in Table 6, using the second-to-last layer’s features from Qwen-VL results in comparable generation performance to umT5, but with a larger model size.

Figure 17: Training loss curve with different text encoders.

Table 6: FID scores for different text encoders. The “last layer” and “second last layer” refer to the text features extracted from the last and second last layers of the encoder, respectively.

Ablation on autoencoder. In addition to our VAE, we design a variant called VAE-D, where the reconstruction loss is replaced by a diffusion loss. We adopt both the pre-trained VAE and VAE-D on the text-to-image generation task for 150,000 training steps until achieving convergence. We measure the FID scores at both 100,000 and 150,000 steps. As shown in Table 5, VAE model consistently achieves lower FID compared to VAE-D.