5 Extended Applications

本文为论文《Wan 2.1：开放先进的视频生成基座模型》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

5.1 Image-to-Video Generation

The image-to-video (I2V) generation task focuses on synthesizing a dynamic video sequence from a static input image guided by a textual prompt. This approach significantly enhances the controllability of video generation by anchoring the output to a specific initial frame, thereby garnering substantial interest within the research community. Existing approaches, such as I2VGen-XL (Zhang et al., 2023c), SVD (Blattmann et al., 2023), and CogVideo (Yang et al., 2025b), extend T2V frameworks to I2V by channel-wise concatenation of the conditional latent representation with the noise latent. Building upon this paradigm, our Wan-I2V model adopts a similar strategy, capitalizing on the robust prior knowledge embedded in our foundational T2V model to achieve high-quality I2V generation.

5.1.1 Model Design

We introduce an additional condition image as the first frame to control video synthesizing. Specifically, we concatenate the condition image $I\in{R^{C\times 1\times H\times W}}$ with zero-filled frames along the temporal axis, and these guidance frames $I_{c}\in{R^{C\times T\times H\times W}}$ are compressed by Wan-VAE into condition latent $z_{c}\in{R^{c\times t\times h\times w}}$ ($c=16$ denotes the latent channel, $t=1+(T-1)/4$, $h=H/8$, $w=W/8$). Additionally, we introduce a binary mask $M\in{0,1}^{1\times T\times h\times w}$ where $1$ indicates the preserved frame and $0$ denotes the frames to be generated. The spatial size of mask $M$ is consistent with condition latent $z_{c}$, but $M$ shares the same temporal length with the target video, which is then rearranged as shape of ${s\times t\times h\times w}$ where $s$ is temporal stride of Wan-VAE. The noise latent $z_{t}$, condition latent $z_{c}$, and rearranged mask $m$ are concatenated along the channel axis and then passed through the DiT model of Wan. Since the input of the I2V DiT model has more channels than T2V (i.e., $2\times c+s$ v.s. $c$), an additional projection layer is employed, which is initialized with zero values. Moreover, we utilize the image encoder of CLIP (Radford et al., 2021) to extract feature representations from the condition image, and the extracted features are projected by a three-layer multi-layer perceptron (MLP), which plays a role as global context. The produced global context is then injected into the DiT model via decoupled cross-attention.

In practice, the aforementioned approach as illustrated in Fig. 18 is extensible to other controllable generation tasks, such as first-last frame transformation and video continuation. The proposed masking mechanism explicitly delineates the input conditions and specifies the frames to be generated. To optimize the efficacy of the masking strategy, we incorporate multiple tasks, i.e., image-to-video generation, video continuation, first-last frame transformation, and random frame interpolation, into our unified framework. Specifically, we adopt the following training paradigm for the I2V model. During the joint training phase, we conduct unified pre-training of our mask-guided model across these diverse tasks. This phase facilitates the model’s ability to discern which positions should be preserved and which should be generated, depending on the input content. Subsequently, in the fine-tuning stage, we concentrate on refining the model’s performance on each individual task.

5.1.2 Dataset

During the joint training phase, we utilize the same training dataset employed for T2V pertaining. This phase equips the model with the foundational capability to generate motion conditioned on reference images. In the fine-tuning phase, we curate task-specific datasets tailored to the unique characteristics of each individual task.

Image-to-video dataset. In the early experiments, we found that when the first frame in the training data significantly differed from the video content, the model struggled to learn stable image-driven video generation. From this perspective, a smaller difference between the first frame and the video content in the training data is beneficial for model training. We calculate the difference between the first frame and the video content based on SigLIP (Zhai et al., 2023) features. Specifically, we compute the cosine similarity between the features of the first frame and the mean of the features of the remaining frames, and then we retain only the videos with a similarity greater than a predefined threshold.

Figure 18: Wan-I2V model framework. The mask mechanism enables the model to selectively focus on input frames such that the proposed framework demonstrates compatibility with video continuation and first-last frame transformation tasks.

Video continuation dataset. Similarly, our findings also indicate that training on temporally consistent videos significantly improves the performance of video continuation tasks. To quantify the consistency between the initial and final segments of a video (i.e., the first 1.5 seconds and the last 3.5 seconds), we compute the cosine similarity of their respective SigLIP features, followed by using the computed similarity scores to filter and select the required training dataset.

First-last frame transformation dataset. Unlike the I2V task, the community places greater emphasis on the smooth transition between the initial and final frames in the mage transition task. To this end, we increase the proportion of data samples with significant transition between the first and last frames in the training dataset.

Figure 19: Videos generated by Wan-I2V model. The results demonstrate that our I2V model effectively captures and replicates the dynamics of real-world scenes.

Figure 20: Videos generated by Wan-I2V model. The results indicate that our I2V model can effectively animate diverse types of images into highly realistic and dynamic videos.

5.1.3 Evaluation

The training of the I2V model, along with first-last frame transformation and video continuation, is structured into two stages. In the initial pre-training stage, we employ the same dataset as that used in the T2V task. Given the slightly larger number of frames involved in this stage, the image encoder branch is excluded, which is motivated by the need to maintain a strong alignment between textual input and visual output, ensuring that the model retains its sensitivity to nuanced textual semantics. Upon the stabilization of the pre-training stage, the model has already manifested a variety of capabilities, including I2V generation, video continuation, and so on. This demonstrates the efficacy of the pre-training process in establishing a robust foundation for both spatial and temporal understanding.

Our empirical observations reveal that relying solely on frame-level information is inadequate for achieving optimal performance in specific tasks, such as image-to-video generation, video continuation, and first-last frame transformation.

This insight indicates that while the pre-trained model offers a foundational framework for frame-based video generation, it lacks the requisite contextual and semantic depth to fully capture the temporal consistency of dynamic video sequences, particularly when the number of conditioning frames is highly limited.

To address this limitation, during the SFT stage, we incorporated an image encoder to extract image features through decoupled cross-attention (Ye et al., 2023) like (Guo et al., 2024a), providing global contextual information to enhance the model’s capacities. We conducted experiments based on two pre-trained checkpoints at 480p and 720p resolutions, covering the training of image-to-video generation, video continuation, and first-last frame transformation tasks.

Similarly to human evaluation for our T2V model, we benchmark our I2V model against state-of-the-art techniques to provide a comprehensive comparison focusing on key aspects such as visual quality, motion quality, and matching. The results shown as Table 7 demonstrate that our model performs favorably across all dimensions evaluated. Fig. 19 and Fig. 20 present additional visualizations of our I2V model, offering a clear illustration of its superior performance in animating images into high-quality videos.

Table 7: Win rate gap of I2V models. The values in the table represent the proportion of instances in which the Wan-I2V model was preferred in pairwise comparisons against other models, relative to the total number of comparisons conducted.

5.2 Unified Video Editing

The foundational pre-trained models for text-to-image and text-to-video generation have facilitated the expansion of various downstream tasks and applications, including repainting (AI, 2022; Zhou et al., 2023), editing (Meng et al., 2021; Brooks et al., 2023; Zhang et al., 2023a; Wang et al., 2023b; Wei et al., 2024b), controllable generation (Zhang et al., 2023b; Jiang et al., 2024; Wang et al., 2024c), customized generation (Chen et al., 2023b; Wei et al., 2025), frame reference generation (Yang et al., 2025b; Guo et al., 2024a), efficient generation (Wang et al., 2023c; Yuan et al., 2024a; Liu et al., 2024b), and ID-referenced video synthesis (Pan et al., 2024; Yuan et al., 2025; Liu et al., 2025a). To enhance task flexibility and minimize the overhead associated with deploying multiple models, researchers have increasingly focused on developing unified model architectures (Tan et al., 2024; Chen et al., 2024), such as ACE (Han et al., 2025; Mao et al., 2025) and OmniGen (Xiao et al., 2024). These unified architectures aim to integrate diverse tasks within a single image model, thereby facilitating the creation of various application workflows while maintaining ease of use. In the domain of video, the collaborative transformations in both temporal and spatial dimensions suggest that leveraging a unified model can unlock limitless possibilities for video creation.

In our previous work, VACE (Jiang et al., 2025), we introduced a unified framework for controllable video generation and editing, addressing tasks such as reference-to-video generation, video-to-video editing, and masked video-to-video editing. Building upon the pre-trained model of Wan, we integrate multiple modalities, including images and videos for editing, references, and masks, into the Video Condition Unit (VCU) to support diverse input formats. VACE employs a concept decoupling strategy that enables the model to identify which aspects should be retained and which should be modified, thereby distinctly separating visual modality information in editing and reference tasks. We offer two training modes. The first mode involves training with the VCU as input, fully fine-tuning the entire Wan model. The second mode utilizes a pluggable Context Adapter structure in a Res-Tuning (Jiang et al., 2023) manner, allowing support for controllable and editing tasks without modifying the base model’s weights but with an increased overall model scale. Leveraging the robust video generation capabilities of Wan, this innovative framework demonstrates significant competitiveness in both quantitative and qualitative evaluations. It facilitates the compositional expansion of fundamental tasks, enabling scenarios such as long video re-rendering. Consequently, the framework offers a versatile and efficient solution for video synthesis, opening new avenues for user-driven video content creation and editing.

Figure 21: Unified controllable generation and editing framework. Frames and masks are tokenized through Concept Decoupling, Context Latent Encode, and Context Embedder. We propose two training strategies for Wan: Fully Fine-tuning and Context Adapter Tuning.

5.2.1 Model Design

The Video Condition Unit (VCU), proposed in VACE, serves as an input paradigm that unifies diverse input conditions into textual inputs, frame sequences, and mask sequences. It can be formally represented as follows:

| | $V=[T;F;M],$ | | (4) |

where $T$ is a text prompt, while $F$ and $M$ are sequences of context video frames ${u_{1},u_{2},...,u_{n}}$ and masks ${m_{1},m_{2},...,m_{n}}$ respectively. Here, $u$ is in RGB space, normalized to $[-1,1]$ and $m$ is binary, in which “1”s and “0”s symbolize where to edit or not. $F$ and $M$ are aligned both in spatial size $h\times w$ and temporal size $n$. Under this paradigm, we employ the previously described Wan-VAE to tokenize the input video frames and mask information into context tokens. As illustrated in Fig. 21 (a), these context tokens are then combined with noisy video tokens to fine-tune the Wan model. Additionally, a Context Adapter Tuning strategy is proposed, which allows context tokens to pass through Context Blocks and be reintegrated into the original DiT blocks. This approach facilitates the seamless integration of contextual information, enhancing the model’s ability to handle diverse editing and generation tasks without altering the base model’s weights.

Figure 22: The visualization results of the proposed VACE.

Figure 23: The visualization results of the proposed VACE.

Before tokenization, pixel-level context frames and masks are preprocessed using a concept decoupling strategy and then encoded into the latent space. As shown in Fig. 21 (b), the concept decoupling strategy explicitly separates the data of different modalities and distributions based on masks into two frame sequences of identical shape: $F_{c}=F\times M$ and $F_{k}=F\times(1-M)$, where $F_{c}$ refers to reactive frames containing all pixels to be modified, while $F_{k}$ denotes inactive frames that retain all pixels to be preserved, named inactive frames. This mechanism ensures clear task definitions and guarantees the model’s convergence across different tasks. $F_{c}$, $F_{k}$ are processed by Wan-VAE and mapped into the same latent space of $X$, maintaining their spatio-temporal consistency. To avoid any mishmash of images and videos, reference images are separately encoded by the Wan-VAE encoder and concatenated back along the temporal dimension, while the corresponding parts need to be removed during decoding. $M$ is directly reshaped and interpolated. After that, $F_{c}$, $F_{k}$, and $M$ are mapped into latent spaces and spatio-temporal aligned with $X$ in the shape of $n^{\prime}\times h^{\prime}\times w^{\prime}$.

Figure 24: The visualization results of advanced video/image editing.

5.2.2 Datasets and Implementation

Data construction. To develop an all-in-one model, the diversity and complexity of data construction must increase. For controllable video generation and editing tasks, input modalities are expanded to include target videos, source videos, local masks, references, and more. Efficient and rapid data acquisition for various tasks necessitates maintaining video quality while performing instance-level analysis and understanding. We begin by performing shot slicing on the video data and initially filtering based on resolution, aesthetic score, and motion amplitude. The first frame is then labeled using RAM (Zhang et al., 2023d) and processed with Grounding DINO (Liu et al., 2023d) for detection, enabling secondary filtering of videos with target areas that are either too small or too large. Additionally, we utilize the propagation operation of SAM2 (Ravi et al., 2025) for video segmentation to obtain instance-level information throughout the video. Based on the segmentation results, we further filter instances temporally by calculating the effective frame ratio according to a mask area threshold. The construction for different tasks must be tailored to each task’s specific characteristics. For detailed information, refer to the work (Jiang et al., 2025).

Implementation details. We train the controllable and editing model, pre-trained on Wan-T2V-14B, to support resolutions up to 720p through a multi-stage training process. Initially, we focus on foundational tasks such as inpainting and extension to complement the pre-trained text-to-video models. This phase incorporates masks and develops contextual generation capabilities in both spatial and temporal dimensions. Subsequently, we expand the task by transitioning from single-input to multiple-input reference frames and from individual to composite tasks. Finally, we enhance the model’s quality by fine-tuning it with higher-quality data and longer sequences.

5.2.3 Evaluation

In Fig. 22 and Fig. 23, we present the results of the Wan single model across various tasks. It is evident that the model achieves a high level of performance in video quality and temporal consistency. Additionally, we demonstrate new applications resulting from the combination of different generative capabilities, as illustrated in Fig. 24 (a). Our model performs effectively on these tasks, showcasing significant potential for capability expansion. The unified controllable generation and editing framework proposed by VACE is also applicable to image generation and editing, as shown in Fig. 24 (b).

5.3 Text-to-Image Generation

Wan is jointly trained on both image and video datasets, equipping it with not only advanced video generation capabilities but also exceptional image synthesis performance. This dual-training strategy facilitates cross-modal knowledge transfer, creating a highly versatile framework that achieves powerful results in both domains. In practice, our model has been trained on an image dataset nearly ten times larger than its video counterpart, allowing for synergistic benefits between image and video generation tasks. This extensive training regimen has led to outstanding performance in various aspects of image synthesis. As demonstrated in Fig. 25, Wan generates high-fidelity images across diverse categories, encompassing artistic text-based visuals, photorealistic portraits, imaginative creative designs, and professional-grade product photography.

Figure 25: Image samples generated from Wan model.

5.4 Video Personalization

Video personalization aims to generate videos that maintain consistent identity with the user-provided reference (Wei et al., 2024a; Li et al., 2024; Yuan et al., 2025). In this section, we integrate advanced personalization techniques into the video generation pipeline, achieving state-of-the-art performance. The subsequent discussion provides a comprehensive elaboration of our approach, encompassing the model architecture, personalization data, and experimental validation.

5.4.1 Model Design

The core techniques behind video personalization tackle two primary challenges: (1) the acquisition of high-fidelity personalized identities and (2) the seamless integration of these identity features into the video generation pipeline.

We start from our Wan-T2V foundation model by first obtaining personalized identity information. Existing video personalization approaches usually rely on ID extractors (e.g., ArcFace (Deng et al., 2019)) or versatile visual extractors (e.g., CLIP (Radford et al., 2021)) to obtain identity information. While they make promising progress, their performance is bottlenecked by the limitations of the feature extractors. For instance, ID extractors mainly focus on features for face recognition but may not capture other critical visual cues such as scars or stickers. Moreover, they are susceptible to low-resolution faces, illumination variations, and partial occlusions (such as glasses, masks, or hair). On the other hand, versatile visual extractors are not specifically tailored to the face domain and tend to focus on coarse-grained semantic information. Therefore, we choose not to rely on any feature extractor to avoid information loss, and our generation process is directly conditioned on the input face image in the latent space of Wan-VAE.

There are many ways to inject the personalized identity information into the video generation process. In our experiments, we found that cross-attention operation is suited to interact with dense representations obtained from identity extractors, while self-attention is more appropriate for modeling data in the same latent space. Fig. 26 illustrates the design of our video personalization approach, which follows a self-attention paradigm. Specifically, in the latent space, we first extend $K$ additional frames prepended to the given video using the segmented face images. These face images are extracted from the paired video using face detection and segmentation, and the facial landmarks are further aligned to a black canvas that has the same size as the video frame. Then, along the channel dimension of this extended video, we concatenate the face images with all-ones masks for the first $K$ frames, and blank images with all-zeros masks for the remaining frames. These concatenated signals serve as the conditions. Finally, the diffusion process is performed on this temporally extended video, conditioned on the channel-wise condition signals in an inpainting fashion. Note that no other modifications are applied to our Wan-T2V model.

During training, we randomly drop a portion of face images in the $K$ extended frames, so as to support $0$ to $K$ reference face(s) video generation. In the inference stage, starting from the pure noise of the extended video, we concatenate the user-provided face images (no more than $K$) to the extended frames in the channel dimension and set the corresponding masks as all-ones. The goal is to reconstruct the face images in the first $K$ frames, and synthesize a new personalized video in the subsequent frames that preserves the provided identity.

Figure 26: The core design of the video personalization approach in the latent space of Wan-VAE.

5.4.2 Dataset

We curate a collection of personalization data from our T2V foundation model’s training dataset through careful filtering and automatic data synthesizing. We first filter out about $\mathcal{O}(100)$M videos using our internal human classifiers and perform face detection at 1 FPS on these videos. A video shall be discarded if more than one face is detected in any frame or if over 10% of the frames have no face detected. Then we calculate ArcFace similarities between consecutive frames and discard videos with low similarities. Afterwards, we perform face segmentation to remove the distracting backgrounds, and perform face landmark detection so as to facilitate canvas alignment during our training. Note that we do not filter out faces with the small areas, because we find that such videos usually contain full-body human figures. Finally, we construct about $\mathcal{O}(10)$M personalized videos, where each video is accompanied by 5 segmented faces in average.

We also resort to automatic data synthesizing to improve facial diversity. Particularly, we randomly select $\mathcal{O}(1)$M personalized videos from above, and adopt Instant-ID (Wang et al., 2024b) to synthesize diverse faces for each video. Notably, we build a text template with over 100 prompts, including anime, line art, cinematic, Minecraft, and so on. Every time we take a random prompt from this template, as well as a random human pose estimated from the rest of the videos, to serve as inputs to Instant-ID. We further measure the ArcFace similarity on the generated faces and filter out those with lower similarities. In total, we gather about $\mathcal{O}(1)$M videos with synthesized faces, which greatly improve the diversity of styles, poses, illuminations, and occlusions on our personalization dataset.

Table 8: Comparison of other leading video personalization methods.

5.4.3 Evaluation

We present our video personalization results in Fig. 27, where the left part represents the input identities and the right part represents the synthesized personalized videos. All the testing images are randomly picked from Pexels11 1

www.pexels.com. We also evaluate our model on an unseen evaluation set and measure the ArcFace similarity between input faces and output personalized videos by sampling faces at 1 FPS from videos. The final similarity scores are averaged from these sampled faces. The corresponding results are illustrated in Table 8, where our approach demonstrates competitive video personalization performance compared to other leading commercial and closed-source Chinese competitors.

Figure 27: Visualization of our video personalization approach.

Figure 28: Framework of video generation guided by camera motion.

5.5 Camera Motion Controllability

The camera motion control module is designed to accurately match the motion and viewpoint of the video by leveraging camera trajectory. Specifically, we utilize the extrinsic parameters $[R,t]\in\mathbb{R}^{3\times 4}$ and intrinsic parameters $K_{f}\in\mathbb{R}^{3\times 3}$ for each frame. Our approach consists of two main components to effectively inject camera motion conditions: camera pose encoder and camera pose adapter, as illustrated in Fig. 28.

Figure 29: Generated video results guided by camera motion.

Camera pose encoder. First, for each pixel in each video frame, we use Plücker coordinates to transform the given extrinsic and intrinsic parameters into a sequence of fine-grained positions $P\in\mathbb{R}^{6\times F\times H\times W}$. To adjust the compression resolution of video latent features, we apply the PixelUnshuffle operation in Pytorch (Paszke et al., 2019) to reduce the spatial resolution of $P$ while increasing the number of channels. Finally, we encode the output using a series of convolutional modules aligned with the number of DiT blocks to extract multi-level camera motion features.

Camera pose adapter. To integrate camera motion features into video latent features, we adopt an adaptive normalization layer. Specifically, we transform the input camera motion feature sequence into scaling factors $\gamma_{i}$ and shifting parameters $\beta_{i}$ using two zero-initialized convolution layers. Then, $\gamma_{i}$ and $\beta_{i}$ are incorporated into each DiT block through a simple linear projection defined by the formula $f_{i}=(\gamma_{i}+1)*f_{i-1}+\beta_{i}$, where $f_{i}$ represents the video latent features at layer $i$.

Regarding the training data, we utilize an advanced camera pose estimation algorithm named VGG-SfM (Wang et al., 2024a) to extract camera trajectories from our training videos that exhibit significant camera motion. This process yields approximately $\mathcal{O}(1)$ thousand video segments. We train our module within our text-to-video generation framework using the Adam optimizer. Fig. 29 presents various example results of videos generated under camera motion guidance.

5.6 Real-time Video Generation

Current video generative methods often require significant computational resources and considerable time to produce even short video clips. Systems relying on high-end hardware remain slow, often taking several minutes to generate just a few seconds of video. While the visual quality of these generated videos has seen substantial improvements in recent years, the extended processing times present a major bottleneck in practical applications and iterative design workflows. In fields where rapid prototyping and real-time adjustments are crucial—such as interactive entertainment, virtual reality, and video production—these long generation times limit creative flexibility and slow down the process of refining visual content.

Moreover, the slow speed of current methods limits their use in dynamic environments where real-time feedback is essential. In contexts like live-streaming or gaming, there’s an increasing demand for systems that can instantly render synthetic scenes and characters in response to user actions or changing inputs. Any delay in generating high-quality video clips can result in a suboptimal user experience, reducing immersion and interactivity. By focusing on real-time generation, Wan seeks to enhance both the performance and applicability of video generation techniques, broadening their potential in fast-paced, interactive environments. Fig. 30 and 31 provide such cases of real-time long video stream generation. We then introduce how to achieve this using our Wan models.

Figure 30: Generated 15 minutes video with 8 A100 GPUs in real-time 8 FPS.

Figure 31: Generated videos with a single RTX 4090 GPU, int8 and TensorRT quantization in real-time 20 FPS.

5.6.1 Method

To construct the real-time generation pipeline, we build upon a previously pre-trained version of the Wan model. This design choice offers two primary benefits. First, starting from a well-trained model significantly accelerates convergence and improves training stability, as the model already possesses a strong initialization. Second, the pre-trained model has already captured valuable knowledge about motion patterns and continuous temporal dynamics, which can be directly inherited by the new pipeline to enhance the smoothness and coherence of generated content. Therefore, we adopt the pre-trained Wan model as the foundation for our real-time pipeline 22 2

Part of this section is introduced in a previously released tech report The Matrix (Feng et al., 2024)..

The pre-trained Wan model is designed to generate fixed-length video clips, typically 5 to 10 seconds long. To transform this into a real-time streaming model capable of generating continuous video streams with no predefined length (potentially infinite), two key modifications are required:

• •

Streaming pipeline adaptation. We replace the static generation process, which produces a fixed-length sequence of video tokens in a single pass, with a streaming mechanism that incrementally generates video tokens. In this streaming setup, video tokens are processed through a denoising queue — each time the oldest token is generated and dequeued, a new token is added to the bottom of the queue, enabling continuous generation without length constraints.

• •

Real-time acceleration. In addition to enabling streaming, we optimize the generation speed to ensure the system meets real-time performance requirements, i.e., the pipeline must generate new frames fast enough to keep up with real-time playback.

In the following sections, we will describe each of these modifications in detail, including their design rationale, technical implementation, and empirical performance.

5.6.2 Streaming Video Generation

Traditional Diffusion Transformer (DiT) (Peebles & Xie, 2023) models are constrained in their ability to generate videos longer than a few seconds, even when employing significant spatial and temporal compression through Variational Autoencoders (VAEs) (Kingma, 2013). This limitation stems primarily from the computational and memory-intensive nature of attention mechanisms over extended temporal sequences. To overcome this challenge, we introduce Streamer, a novel approach that leverages a sliding temporal window to manage temporal dependencies efficiently, enabling the generation of long or even infinite-length videos in a streaming fashion.

Shift window denoise process models.

The key innovation of Streamer lies in its assumption that temporal dependencies are confined within a limited time window. By focusing attention computations within this window, Streamer significantly reduces computational overhead while maintaining video continuity. Specifically, Streamer processes video tokens in a queue, denoising them over multiple noise levels within the window. After a fixed number of denoising steps, the leftmost token (with the lowest noise level) is dequeued and cached, while a new token with Gaussian noise is appended to the rightmost position. This sliding mechanism ensures that the window maintains a consistent length while facilitating the generation of continuous video frames.

Training and inference.

Streamer is fine-tuned from a pre-trained DiT model. During training, a sequence of $2w$ video tokens is sampled, where $w$ is the size of the sliding window (typically set to $w=T$, the number of diffusion solver steps). The first $w$ tokens are used solely for ‘warming up’ the model and do not contribute to the loss computation. The loss is calculated only on the last $w$ tokens, ensuring that the model learns to generate coherent video frames within the window. At inference time, the same warmup strategy is employed: the first $w$ tokens are discarded, and the generated video begins from the $(w+1)$-th token.

Ensuring continuity during training and inference.

To maintain smooth transitions between consecutive windows, cached tokens are re-introduced into the token queue at a noise level of 0. This allows previously generated tokens to participate in the denoising process of subsequent windows, ensuring temporal consistency and continuity in the generated video.

Streamer represents a significant advancement in video generation by adapting DiT models for streaming applications. Its sliding temporal window approach addresses the limitations of traditional DiT models, enabling the efficient generation of infinite-length videos while maintaining high quality and temporal coherence. This method opens new possibilities for real-time video generation and long-form content creation. We summarize its advantages as follows:

• •

Infinite-Length Video Generation: By leveraging a sliding window mechanism, Streamer can generate videos of arbitrary length without incurring prohibitive computational costs;

• •

Efficient Attention Computation: Limiting attention computations to a confined temporal window reduces memory and processing demands;

• •

Seamless Continuity: The caching and re-introduction of tokens ensure smooth transitions between windows, preserving video quality over extended durations.

5.6.3 Consistency Model Distillation

After successfully extending the DiT model to Streamer for infinite-length video generation, the next critical step is to address the challenge of achieving real-time rendering of the simulated world. While Streamer enables the generation of long or even infinite videos by efficiently managing temporal dependencies within a sliding window, the computational demands of the diffusion process still pose a bottleneck for real-time applications. To overcome this, we propose a novel integration of Streamer with Consistency Models, a cutting-edge approach for accelerating diffusion-based generative processes. Specifically, we leverage the Latent Consistency Model (LCM (Luo et al., 2023)) and its video version (VideoLCM (Wang et al., 2023c)), which distill the original diffusion process and class-free guidance into a highly efficient four-step consistency model. This distillation process significantly reduces the number of sampling steps required for high-quality generation while maintaining the temporal coherence and streaming capabilities inherent to Streamer.

The integration of LCM with Streamer is designed to preserve the denoising window mechanism, ensuring that the sliding temporal window continues to manage dependencies effectively. During training, we optimize this combined framework to balance efficiency and quality. The result is a 10 - 20 $\times$ acceleration in inference speed, enabling the model to achieve a rendering rate of 8 - 16 FPS. This dramatic improvement in performance makes the system viable for real-time applications, such as interactive simulations, live video synthesis, and dynamic content generation in virtual environments. By combining the strengths of Streamer—its ability to generate infinite-length videos with temporal coherence—and LCM—its efficiency in accelerating the diffusion process—we bridge the gap between high-quality video generation and real-time responsiveness. This advancement not only enhances the practicality of diffusion models for streaming applications but also opens up new possibilities for immersive, interactive, and real-time media creation.

Quantization for running in customer-level devices.

While the generating speed has been accelerated to real-time levels, deploying the model on consumer-level devices remains challenging due to the high computational and memory requirements, even for high-end GPUs like the NVIDIA 4090. To address this, we introduce quantization techniques to optimize the model for efficient deployment. Specifically, we employ two distinct quantization strategies: int8 quantization (torchao maintainers & contributors, 2024) for attention layers and linear heads, and TensorRT quantization (NVIDIA, 2023) for the overall model. The int8 quantization approach significantly reduces memory consumption by converting weights and activations to 8-bit integers, while maintaining the overall generation quality. However, this method provides minimal acceleration in generating speed, as it primarily focuses on memory efficiency rather than computational optimization. On the other hand, TensorRT quantization offers a more comprehensive solution, enabling substantial acceleration in generation speed. This technique allows the model to achieve real-time performance of 8 FPS even on a single 4090 GPU, making it feasible for consumer-level devices. However, TensorRT quantization introduces a modest increase in network error, which can be observed even when its built-in error-checking mechanisms are enabled. This error manifests as slight deviations in output quality, such as minor artifacts or inconsistencies in the generated video. In some cases, it may also lead to measurable instability, such as flickering or temporal incoherence. Despite these trade-offs, the combination of int8 and TensorRT quantization provides a balanced approach to optimizing the model for real-time, consumer-level deployment, ensuring both efficiency and practicality while maintaining acceptable generation quality. By carefully tuning the quantization parameters and leveraging TensorRT’s error-checking features, we mitigate these issues to the greatest extent possible, enabling robust and efficient performance on consumer-grade hardware.

5.7 Audio Generation

The primary objective of audio generation in this work is to produce synchronized soundtracks for video clips, formulated as a video-to-audio (V2A) generation framework. The generated soundtracks consist of ambient sound and background music, explicitly excluding speech or vocal elements. In contrast to text-to-audio models (Liu et al., 2023b; Liu et al., 2024c), the ambient sound generated by video-to-audio models must be temporally aligned with the visual content of the video, while the accompanying music should accurately reflect the emotional tone and contextual setting of the video, as would naturally be expected. Furthermore, to enhance user control over sound design, our framework integrates both video and textual prompts, enabling users to specify on-screen or off-screen sounds and define the style and presence of background music.

5.7.1 Model Design

Consistent with our video generation framework, our video-to-audio model also utilizes a diffusion transformer (DiT) (Peebles & Xie, 2023) based on the flow-matching (Lipman et al., 2022) to model the denoising diffusion process in the audio domain. The detailed pipeline of our V2A model is illustrated in Fig. 32.

Figure 32: Framework of video-to-audio generation. Our model processes both a video chunk and its associated textual description as dual inputs to synthesize high-quality, semantically coherent audio.

Figure 33: Audio generation samples of MMAudio and ours.

Audio autoencoder. A widely adopted audio compression approach involves transforming raw waveforms into mel-spectrograms, followed by encoding them into $H^{a}\times W^{a}\times C^{a}$ latent features using an image-based variational autoencoder, as demonstrated in (Liu et al., 2023b; Liu et al., 2024c). However, in the context of a diffusion transformer backbone, these latent features must be partitioned into patches and reshaped into a tensor of size $(H^{a}\cdot W^{a})\times C^{a}$. This patching-and-reshaping process can disrupt the temporal alignment with the corresponding video content, posing a significant challenge for synchronization. Considering this limitation, we train a compression model that operates directly on raw waveforms. Our 1D-VAE generates latent features of size $T^{a}\times C^{a}$, where $T^{a}$ represents the sequence length along the time axis, preserving explicit temporal information essential for accurate synchronization.

Video and text encoder. To achieve frame-accurate synchronization between synthesized audio and visual sequences, we establish temporal coherence through multimodal feature fusion. Our architecture first extracts frame-wise visual embeddings using a CLIP Radford et al. (2021) model, then performs temporal rate adaptation through feature replication to match the audio feature’s sampling rate, following (Polyak et al., ). The synchronized visual and acoustic features undergo dimension transformation via linear projection layers to align with DiT’s latent space, followed by element-wise summation for multimodal fusion. For linguistic understanding across languages, we leverage the pre-trained umT5 model (Chung et al., 2023) as our text encoder, capitalizing on its cross-lingual representation capabilities through frozen parameters.

Data. The training data for our V2A model is derived from the video generation dataset through a rigorous filtering process. We systematically remove videos lacking soundtracks or containing speech/vocal music, resulting in a refined subset of $\mathcal{O}(1)$ thousand hours. To enhance multimodal representation, we employ a comprehensive captioning strategy: dense video descriptions are complemented with audio-specific captions generated by Qwen2-audio (Chu et al., 2024). These audio captions are categorized into ambient sounds and musical compositions, with the latter characterized by style, rhythm, melody, and instrumentation. The final structured caption integrates three components: (1) dense video description, (2) ambient sound characterization, and (3) background music analysis, providing a unified multimodal representation for training.

Implementation details. Our V2A architecture generates high-fidelity stereo audio with a maximum duration of 12 seconds at a 44.1 kHz sampling rate. The model processes multimodal inputs through dedicated encoders: umT5-XXL produces 4096-dimensional text embeddings, while the CLIP extracts 1024-dimensional visual embeddings per frame. The audio and video features are projected into a unified 1536-dimensional latent space within the DiT backbone. For temporal alignment, we downsample input videos and yield precisely 48 frames for 12-second clips, ensuring frame-accurate synchronization between visual and audio modalities. The system processes latent sequences of length 256 in one batch. To enhance the model’s capability of generating audio solely from visual cues, we implement a random masking strategy during training where ambient sound and music captions are selectively omitted with a predefined probability. This conditioning mechanism forces the model to establish robust cross-modal associations between visual content and corresponding audio patterns, while maintaining the flexibility to utilize textual descriptions when available.

5.7.2 Evaluation

We provide qualitative evaluations of audio generation in Fig. 33, where we compare our method with the recently released open-sourced approach MMAudio (Cheng et al., 2024). This comparative analysis is particularly meaningful as MMAudio represents the most similar and competitive baseline to our method, demonstrating impressive performance in audio generation tasks. The videos are generated by our text-to-video model with a duration of 5 seconds.

Our V2A model demo

（本文超出站点单文档体积上限，已截断）