3 Data Processing Pipeline

本文为论文《Wan 2.1：开放先进的视频生成基座模型》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

High-quality data is essential for training large generative models, and an automated data construction pipeline significantly enhances the efficiency of the training process. In developing our dataset, we prioritized three core principles: high quality, high diversity, and substantial scale. Following these principles, we curated a dataset comprising billions of videos and images. This section offers a detailed introduction to the data construction pipeline employed for Wan.

3.1 Pre-training Data

We curated and deduplicated a candidate dataset sourced from both internal copyrighted sources and publicly accessible data. In the pre-training stage, our goal is to select high-quality and diverse data from this expansive yet noisy dataset to facilitate effective training. Throughout the data mining process, we designed a four-step data cleaning process, focusing on fundamental dimensions, visual quality, and motion quality. Subsequently, we will also highlight our data processing workflow for constructing visual text data.

Fundamental dimensions. The fundamental dimensions of our data filtering framework focus on the intrinsic attributes of the source video and image data, enabling efficient preliminary filtering out of all the unsuitable data. Specifically, our multidimensional filtering approach encompasses the following critical aspects: Text detection. A lightweight OCR detector is implemented to quantify text coverage ratios, effectively excluding videos and images with excessive textual elements to maintain visual clarity. Aesthetic evaluation: We use the widely adopted LAION-5B (Schuhmann et al., 2021) aesthetic classifier to perform an initial quality assessment of our images, quickly filtering out low-quality data. NSFW Score. Through our internal safety assessment model, we systematically evaluate and filter inappropriate content based on computed NSFW scores in all training data. Watermark and logo detection. We detect whether the video or images contain watermarks and logos, and crop these elements during training. Black border detection. Utilizing heuristic-based detection methods, we automatically crop extraneous black borders to maintain focus on content-rich regions. Overexposure detection. Our trained expert classifier evaluates and filters out data with abnormal tonal distributions, ensuring optimal visual quality in the training dataset. Synthetic image detection. Empirical evidence indicates that even minimal contamination ($ Visual quality. This primarily involves selecting data from the candidate dataset that is of relatively high quality and meets pre-training standards. This subset of data must have good visual quality and its overall distribution should align with that of natural data. In this process, we divide the task into two parts: clustering and scoring. Clustering is used to split all the data into smaller subsets, allowing us to process each subset individually. The benefit of this approach is that it prevents the loss of small yet important data segments that might occur due to a long-tail distribution, thereby maintaining the original data distribution. Specifically, we divide the data into 100 clusters and then select a certain amount of data from each cluster for the next stage of data processing. Scoring is used to assign a quality score to each video or image, facilitating the selection and processing of data at different stages. Specifically, we select samples from each cluster for manual scoring (ranging from 1 to 5, with 1 being the worst and 5 being the best). We then train an expert assessment model using all the annotated data to score the entire dataset.

Motion quality. The goal of motion quality assessment is to select videos that are natural, complete, and with significant motion, while avoiding static or jittery movements. We classify video data into six distinct motion quality tiers. Optimal motion: This level represents videos with optimal attributes, such as significant motion layout, perspective, and amplitude, along with clean, smooth movement. Medium-quality motion: Videos in this category exhibit noticeable movements but may have minor issues like multiple subjects or partial occlusion. This data ensures motion diversity and can be used during pre-training to help the model better understand spatio-temporal relationships. Static Videos: This category primarily focuses on the videos, which consist largely of chat and interview-style videos. While these videos contain minimal motion information, they are of high quality. Therefore, we need to identify them separately and reduce their sampling ratio. Camera-driven Motion: Footage dominated by camera movement (e.g., aerial shots) with minimal subject motion. Given their similarity to static images, these receive substantially lower sampling priority. Low-quality Motion: Videos exhibiting excessive subjects, severe occlusions, or unclear main subjects (e.g., crowded street scenes). Such content is excluded due to its negative impact on training efficiency and motion generation quality. Shaky camera footage: Amateur recordings with pronounced camera shake, often causing motion blur and ambiguous foreground-background differentiation. This category is systematically excluded from training consideration.

Visual text data. In addition, we also introduce a novel data processing approach to enhance the visual text generation of Wan. This approach comprises two distinct processing branches designed to improve both the accuracy and the harmony of text rendering. On one hand, we synthesize hundreds of millions of text-containing images by rendering Chinese characters on a pure white background. On the other hand, we collect large amounts of text-containing images from vast real-world data. We employ multiple OCR models to accurately recognize both English and Chinese text from images and videos, and subsequently input these extracted text contents into the multimodal language model Qwen2-VL. This process generates natural descriptions of the images, ensuring that the descriptions incorporate as much precise text content as possible. We collect a mass of real-world image-text pairs by leveraging this processing pipeline. By integrating both synthetic and real data during the pre-training stage, our approach effectively generalizes to generate rare words in videos with accurate glyphs and high realism, demonstrating significant superiority in visual text generation.

Figure 3: Data provisioning across different training phases. For each stage, we dynamically adjust the proportions of data related to motion, quality, and category based on data throughput.

3.2 Post-training Data

The core objective of post-training is to improve both the visual fidelity and motion dynamics of generated videos through high-quality data. Our data processing pipeline in this stage employs distinct strategies for static and dynamic data: image data undergoes optimization for improved visual quality, while video data is specifically processed to refine motion quality.

Image processing. From the pool of high-scoring image data, we perform an additional refinement process to identify top-quality samples that the images excel in aspects such as quality, composition and details. Specifically, we construct the curated dataset using two methods: expert-based collection and manual collection. The former method involves selecting the top 20% of images based on scores predicted by an expert model. For this subset, we also consider factors such as style and category to ensure diversity in data distribution. The latter approach involves manually collecting top-quality images from various categories and data sources while also filling in any missing concepts in our dataset to enhance the model’s generalization capabilities. Through this process, we collected a total of millions of curated images.

Video processing. In this phase, we employ a strategy similar to the one used for images to collect top-quality videos. First, we filter out some top-ranked videos from the candidate datasets using the visual quality classifier. Then, based on the motion quality classifier, we select millions of videos featuring simple movements and millions of videos with complex movements. All video selections follow a strategy that emphasizes category balance and high diversity. Meanwhile, we select data from 12 major categories, including technology, animals, arts, humans, vehicles, and etc, in order to enhance the model’s generative capabilities for commonly used categories.

3.3 Dense Video Caption

While our dataset includes text descriptions from original webpages for many images and videos, they are often too simplistic to convey detailed visual content. DALL-E 3 (Betker et al., 2023) has demonstrated that the ability of visual generation models to follow prompts can be significantly enhanced by training on highly descriptive generated visual captions. Therefore, we develop an internal caption model to generate dense captions for each image and video of our dataset. To train this model, we incorporate both open-source vision-language datasets and additional data collected in-house.

3.3.1 Open Source Dataset

We collect widely used vision-language datasets for both images and videos. Our collection includes not only caption datasets but also visual Q&A datasets focused on visual content, such as actions, counting, and OCR. In some scenarios, we require the caption model to generate captions in a specific style or content based on user instructions. Therefore, we also collected pure text instruction data to enhance our model’s ability to follow instructions.

3.3.2 In-house Dataset

We collect an in-house dataset for various tasks to enhance the model’s capabilities in specific areas.

Celebrities, landmarks, and movie characters. To train our model to recognize celebrities, landmarks, and characters, we collect dataset comprising thousands of identities. First, we use large language models (LLMs) to collect names, then retrieve corresponding images from our image-text database using a CLIP-style model. We test various CLIP-style models and find that TEAM (Xie et al., 2022) excels in recognizing individuals, particularly Chinese celebrities. To further reduce noise in our dataset, we implemented keyword matching for the descriptions of the image-text pairs. An image is retained only if it is retrieved by the TEAM model and matches the specified keywords.

Object counting. To enhance our model’s visual counting ability, we compiled a counting dataset by retrieving images with text descriptions that include words like “one,” “two,” “three”, etc. These numbers provide coarse annotations for the images. We then use a LLM to extract (category, count) pairs from the corresponding text. Finally, we use Grounding DINO (Liu et al., 2023d) to count objects in those categories. An image is retained only if its text description matches the results from Grounding DINO (Liu et al., 2023d).

OCR. To improve our model’s capability in generating text descriptions, we collect an OCR-augmented image caption dataset. Initially, we used an off-the-shelf OCR detector to extract text from images. Then, we prompt our caption model to generate descriptions based on these OCR results. This approach ensures accurate text descriptions by using the extracted OCR text as prior knowledge. Currently, we include only English and Chinese texts.

Camera angle and motion. We observe that current multi-modal language models (MLLMs) struggle with predicting camera angles and motion, including state-of-the-art commercial models like GPT-4o and Google Gemini Pro. To enhance these predictions, we first annotate a set of videos for camera angle and motion. This dataset can be utilized in two ways: directly training our caption model used in the final training stage, or by training an expert model to annotate more video data, aiding our model during the early training phase. This dataset enhances the accuracy of our motion and angle descriptions, thereby improving the camera motion controllability of our video generation model.

Fine-grained categories. To enhance our model’s ability to recognize fine-grained categories such as animals, plants, and vehicles, we created a dataset containing several million images across these categories.

Relational understanding. We enhance our model’s relational understanding by gathering dataset that focus on spatial relationships, such as left, right, up, and down, through the incorporation of existing object detection datasets.

Re-caption. An important capability of a caption model is to leverage existing tags or brief captions to generate dense captions. To facilitate this, we curate a re-caption dataset that extends brief text tags or captions to detailed descriptions based on image content.

Editing instruction caption. We collect a dataset that describes the differences or transformations between two images. This type of caption is valuable for image editing tasks.

Group image description. We collect a dataset that provides descriptions for groups of images. Each caption begins with a description of the common features shared by the images, followed by individual descriptions of each image, using similar phrasing.

Human-annotated image and video captions. We gather human-annotated dense caption for both images and videos. These represent our highest-quality caption data and are used in the final stage of model training.

Figure 4: Our caption model achieves overall performance comparable to Google Gemini 1.5 Pro. In particular, it excels in areas such as events, camera angle, camera motion, style, and color.

3.3.3 Model Design

Architecture. Our caption model adopts a LLaVA-style architecture (Liu et al., 2023c). We use a ViT encoder to extract visual embeddings of images and video frames. These embeddings are projected through a two-layer perception and then fed into the Qwen LLM (QwenTeam, 2024).

For the image input, we adopt dynamic high resolution as in LLaVA. An image can be divided into a maximum of seven patches. For each patch, we adaptively pool the vision embeddings to a 12$\times$12 grid representation to reduce computation.

For the video, we sample 3 frames per second, with an upper limit of 129 frames. To further reduce computation, we employ a slow-fast encoding strategy: maintaining the original resolution for every fourth frame and applying global average pooling to the embeddings of the remaining frames. Experiments on long video benchmarks indicate that the slow-fast mechanism enhances understanding with a limited number of visual tokens (e.g., improving performance from 67.6% to 69.1% on VideoMME (Fu et al., 2024)) without subtitles.

Training. Following recent state-of-the-art methods, our training process is divided into three stages. In the first stage, we freeze the ViT and LLM, and train only the multi-layer perceptron to align the visual embeddings with the LLM input space, using a learning rate of 1e-3. In the second stage, all parameters are made trainable. In the final stage, we conduct end-to-end training on a small set of high-quality data. For the last two stages, the learning rates are set to 1e-5 for the LLM and MLP, and 1e-6 for the ViT.

3.3.4 Evaluation

Automatic evaluation is crucial for the development of our caption model, allowing us to identify the strengths and weaknesses of each model version. Following CAPability (Liu et al., 2025b), we develop an automatic caption evaluation pipeline. We focus on ten key visual dimensions in video generation: action, camera angle, camera motion, object category, object color, object counting, OCR, scene, style, and event. For each dimension, we randomly sampled 1,000 videos and their corresponding captions generated by our model. We also used Google Gemini 1.5 Pro to generate dense video captions for this dataset. We evaluate our captions and Gemini-generated captions by calculating the F1 metrics for each dimension following CAPability (Liu et al., 2025b). The results are illustrated in Fig. 4. Our caption performs better in video event, camera angle, camera motion, style, and object color, while Gemini excels in action, OCR, scene, object category, and object counting.