3 Experiments

本文为论文《Janus-Pro：统一多模态理解与生成》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

3.1 Implementation Details

In our experiments, we utilize DeepSeek-LLM ($1.5$B and $7$B) [3] with a maximum supported sequence length of $4096$ as the base language model. For the vision encoder used in understanding tasks, we select SigLIP-Large-Patch$16$-$384$ [53]. The generation encoder has a codebook of size $16,384$ and downsamples images by a factor of $16$. Both the understanding adaptor and the generation adaptor are two-layer MLPs. The detailed hyperparameters for each stage are provided in Table 2. Please note that for Stage II, we employ an early stopping strategy, halting at 270K steps. All images are resized to $384\times 384$ pixels. For multimodal understanding data, we resize the long side of the image and pad the short side with the background color (RGB: $127$, $127$, $127$) to reach $384$. For visual generation data, the short side is resized to $384$, and the long side is cropped to $384$. We use sequence packing during training to improve training efficiency. We mix all data types according to the specified ratios in a single training step. Our Janus-Pro is trained and evaluated using HAI-LLM [15], which is a lightweight and efficient distributed training framework built on top of PyTorch. The whole training process took about $9$/$14$ days on a cluster of $16$/$32$ nodes for $1.5$B/$7$B model, each equipped with $8$ Nvidia A$100$ ($40$GB) GPUs.

Table 3: Comparison with state-of-the-arts on multimodal understanding benchmarks. “Und.” and “Gen.” denote “understanding” and “generation”, respectively. Models using external pretrained diffusion model are marked with †.

3.2 Evaluation Setup

Multimodal Understanding. To assess multimodal understanding capabilities, we evaluate our model on widely recognized image-based vision-language benchmarks, which include GQA [17], POPE [23], MME [12], SEED [21], MMB [29], MM-Vet [51], and MMMU [52].

Visual Generation. For evaluating visual generation capabilities, we use GenEval [14] and DPG-Bench [16]. GenEval is a challenging benchmark for text-to-image generation, designed to reflect the comprehensive generative abilities of visual generation models by offering a detailed instance-level analysis of their compositional capabilities. DPG-Bench (Dense Prompt Graph Benchmark) is a comprehensive dataset consisting of 1065 lengthy, dense prompts, designed to assess the intricate semantic alignment capabilities of text-to-image models.

3.3 Comparison with State-of-the-arts

Multimodal Understanding Performance. We compare the proposed method with state-of-the-art unified models and understanding-only models in Table 3. Janus-Pro achieves the overall best results. This can be attributed to decoupling the visual encoding for multimodal understanding and generation, mitigating the conflict between these two tasks. When compared to models with significantly larger sizes, Janus-Pro remains highly competitive. For instance, Janus-Pro-7B outperforms TokenFlow-XL ($13$B) on all benchmarks except GQA.

Table 4: Evaluation of text-to-image generation ability on GenEval benchmark. “Und.” and “Gen.” denote “understanding” and “generation”, respectively. Models using external pretrained diffusion model are marked with †.

Table 5: Performances on DPG-Bench. The methods in this table are all generation-specific models except Janus and Janus-Pro.

Visual Generation Performance. We report visual generation performance on GenEval and DPG-Bench. As shown in Table 4, our Janus-Pro-7B obtains $80$% overall accuracy on GenEval, which outperforms all the other unified or generation-only methods, e.g., Transfusion [55] ($63$%) SD3-Medium ($74$%) and DALL-E $3$ ($67$%). This demonstrates that our approach has better instruction-following capabilities. As shown in Table 5, Janus-Pro achieves a score of $84.19$ on DPG-Bench, surpassing all other methods. This demonstrates that Janus-Pro excels in following dense instructions for text-to-image generation.

3.4 Qualitative Results

We present results on multimodal understanding in Figure 4. Janus-Pro exhibits impressive comprehension abilities when handling inputs from various contexts, showcasing its powerful capabilities. We also present some text-to-image generation results in the lower part of Figure 4. The images generated by Janus-Pro-7B are highly realistic, and despite having a resolution of only $384\times 384$, they still contain a lot of details. For imaginative and creative scenes, Janus-Pro-7B accurately captures the semantic information from the prompts, producing well-reasoned and coherent images.

Figure 4: Qualitative results of multimodal understanding and visual generation capability. The model is Janus-Pro-7B and the image output resolution of visual generation is $384\times 384$. Best viewed on screen.