References

本文为论文《Janus-Pro：统一多模态理解与生成》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

  • [1]

J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou.

Qwen-vl: A frontier large vision-language model with versatile abilities.

arXiv preprint arXiv:2308.12966, 2023.

  • [2]

J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo, et al.

Improving image generation with better captions.

Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023.

  • [3]

X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, et al.

Deepseek llm: Scaling open-source language models with longtermism.

arXiv preprint arXiv:2401.02954, 2024.

  • [4]

J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Y. Wu, Z. Wang, J. Kwok, P. Luo, H. Lu, et al.

Pixart-$alpha$: Fast training of diffusion transformer for photorealistic text-to-image synthesis.

arXiv preprint arXiv:2310.00426, 2023.

  • [5]

J. Chen, C. Ge, E. Xie, Y. Wu, L. Yao, X. Ren, Z. Wang, P. Luo, H. Lu, and Z. Li.

PixArt-Sigma: Weak-to-strong training of diffusion transformer for 4K text-to-image generation.

arXiv preprint arXiv:2403.04692, 2024.

  • [6]

X. Chu, L. Qiao, X. Lin, S. Xu, Y. Yang, Y. Hu, F. Wei, X. Zhang, B. Zhang, X. Wei, et al.

Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices.

arXiv preprint arXiv:2312.16886, 2023.

  • [7]

X. Chu, L. Qiao, X. Zhang, S. Xu, F. Wei, Y. Yang, X. Sun, Y. Hu, X. Lin, B. Zhang, et al.

Mobilevlm v2: Faster and stronger baseline for vision language model.

arXiv preprint arXiv:2402.03766, 2024.

  • [8]

W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi.

Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.

  • [9]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.

Imagenet: A large-scale hierarchical image database.

In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.

  • [10]

R. Dong, C. Han, Y. Peng, Z. Qi, Z. Ge, J. Yang, L. Zhao, J. Sun, H. Zhou, H. Wei, et al.

Dreamllm: Synergistic multimodal comprehension and creation.

arXiv preprint arXiv:2309.11499, 2023.

  • [11]

P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, D. Podell, T. Dockhorn, Z. English, K. Lacey, A. Goodwin, Y. Marek, and R. Rombach.

Scaling rectified flow transformers for high-resolution image synthesis, 2024.

URL https://arxiv.org/abs/2403.03206.

  • [12]

C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, et al.

Mme: A comprehensive evaluation benchmark for multimodal large language models.

arXiv preprint arXiv:2306.13394, 2023.

  • [13]

Y. Ge, S. Zhao, J. Zhu, Y. Ge, K. Yi, L. Song, C. Li, X. Ding, and Y. Shan.

Seed-x: Multimodal models with unified multi-granularity comprehension and generation.

arXiv preprint arXiv:2404.14396, 2024.

  • [14]

D. Ghosh, H. Hajishirzi, and L. Schmidt.

Geneval: An object-focused framework for evaluating text-to-image alignment.

Advances in Neural Information Processing Systems, 36, 2024.

  • [15]

High-flyer.

Hai-llm: Efficient and lightweight training tool for large models, 2023.

URL https://www.high-flyer.cn/en/blog/hai-llm.

  • [16]

X. Hu, R. Wang, Y. Fang, B. Fu, P. Cheng, and G. Yu.

Ella: Equip diffusion models with llm for enhanced semantic alignment.

arXiv preprint arXiv:2403.05135, 2024.

  • [17]

D. A. Hudson and C. D. Manning.

Gqa: A new dataset for real-world visual reasoning and compositional question answering.

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.

  • [18]

Y. Jin, K. Xu, L. Chen, C. Liao, J. Tan, B. Chen, C. Lei, A. Liu, C. Song, X. Lei, et al.

Unified language-vision pretraining with dynamic discrete visual tokenization.

arXiv preprint arXiv:2309.04669, 2023.

  • [19]

H. Laurençon, D. van Strien, S. Bekman, L. Tronchon, L. Saulnier, T. Wang, S. Karamcheti, A. Singh, G. Pistilli, Y. Jernite, and et al.

Introducing idefics: An open reproduction of state-of-the-art visual language model, 2023.

URL https://huggingface.co/blog/idefics.

  • [20]

H. Laurençon, A. Marafioti, V. Sanh, and L. Tronchon.

Building and better understanding vision-language models: insights and future directions., 2024.

  • [21]

B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan.

Seed-bench: Benchmarking multimodal llms with generative comprehension.

arXiv preprint arXiv:2307.16125, 2023a.

  • [22]

D. Li, A. Kamko, E. Akhgari, A. Sabet, L. Xu, and S. Doshi.

Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation.

arXiv preprint arXiv:2402.17245, 2024a.

  • [23]

Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen.

Evaluating object hallucination in large vision-language models.

arXiv preprint arXiv:2305.10355, 2023b.

  • [24]

Z. Li, H. Li, Y. Shi, A. B. Farimani, Y. Kluger, L. Yang, and P. Wang.

Dual diffusion for unified image generation and understanding.

arXiv preprint arXiv:2501.00289, 2024b.

  • [25]

Z. Li, J. Zhang, Q. Lin, J. Xiong, Y. Long, X. Deng, Y. Zhang, X. Liu, M. Huang, Z. Xiao, et al.

Hunyuan-DiT: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding.

arXiv preprint arXiv:2405.08748, 2024c.

  • [26]

H. Liu, C. Li, Y. Li, and Y. J. Lee.

Improved baselines with visual instruction tuning.

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024a.

  • [27]

H. Liu, C. Li, Q. Wu, and Y. J. Lee.

Visual instruction tuning.

Advances in neural information processing systems, 36, 2024b.

  • [28]

H. Liu, W. Yan, M. Zaharia, and P. Abbeel.

World model on million-length video and language with ringattention.

arXiv preprint arXiv:2402.08268, 2024c.

  • [29]

Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al.

Mmbench: Is your multi-modal model an all-around player?

arXiv preprint arXiv:2307.06281, 2023.

  • [30]

Y. Ma, X. Liu, X. Chen, W. Liu, C. Wu, Z. Wu, Z. Pan, Z. Xie, H. Zhang, X. yu, L. Zhao, Y. Wang, J. Liu, and C. Ruan.

Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation, 2024.

  • [31]

mehdidc.

Yfcc-huggingface.

https://huggingface.co/datasets/mehdidc/yfcc15m, 2024.

  • [32]

D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach.

Sdxl: Improving latent diffusion models for high-resolution image synthesis.

arXiv preprint arXiv:2307.01952, 2023.

  • [33]

D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach.

SDXL: Improving latent diffusion models for high-resolution image synthesis.

2024.

  • [34]

L. Qu, H. Zhang, Y. Liu, X. Wang, Y. Jiang, Y. Gao, H. Ye, D. K. Du, Z. Yuan, and X. Wu.

Tokenflow: Unified image tokenizer for multimodal understanding and generation.

arXiv preprint arXiv:2412.03069, 2024.

  • [35]

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen.

Hierarchical text-conditional image generation with clip latents.

arXiv preprint arXiv:2204.06125, 1(2):3, 2022.

  • [36]

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.

High-resolution image synthesis with latent diffusion models.

2022a.

  • [37]

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.

High-resolution image synthesis with latent diffusion models.

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022b.

  • [38]

P. Sun, Y. Jiang, S. Chen, S. Zhang, B. Peng, P. Luo, and Z. Yuan.

Autoregressive model beats diffusion: Llama for scalable image generation.

arXiv preprint arXiv:2406.06525, 2024.

  • [39]

Q. Sun, Q. Yu, Y. Cui, F. Zhang, X. Zhang, Y. Wang, H. Gao, J. Liu, T. Huang, and X. Wang.

Generative pretraining in multimodality.

arXiv preprint arXiv:2307.05222, 2023.

  • [40]

C. Team.

Chameleon: Mixed-modal early-fusion foundation models.

arXiv preprint arXiv:2405.09818, 2024.

  • [41]

G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al.

Gemini: a family of highly capable multimodal models.

arXiv preprint arXiv:2312.11805, 2023.

  • [42]

S. Tong, D. Fan, J. Zhu, Y. Xiong, X. Chen, K. Sinha, M. Rabbat, Y. LeCun, S. Xie, and Z. Liu.

Metamorph: Multimodal understanding and generation via instruction tuning.

arXiv preprint arXiv:2412.14164, 2024.

  • [43]

Vivym.

Midjourney prompts dataset.

https://huggingface.co/datasets/vivym/midjourney-prompts, 2023.

Accessed: [Insert Date of Access, e.g., 2023-10-15].

  • [44]

C. Wang, G. Lu, J. Yang, R. Huang, J. Han, L. Hou, W. Zhang, and H. Xu.

Illume: Illuminating your llms to see, draw, and self-enhance.

arXiv preprint arXiv:2412.06673, 2024a.

  • [45]

X. Wang, X. Zhang, Z. Luo, Q. Sun, Y. Cui, J. Wang, F. Zhang, Y. Wang, Z. Li, Q. Yu, et al.

Emu3: Next-token prediction is all you need.

arXiv preprint arXiv:2409.18869, 2024b.

  • [46]

C. Wu, X. Chen, Z. Wu, Y. Ma, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, C. Ruan, et al.

Janus: Decoupling visual encoding for unified multimodal understanding and generation.

arXiv preprint arXiv:2410.13848, 2024a.

  • [47]

S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua.

Next-gpt: Any-to-any multimodal llm.

arXiv preprint arXiv:2309.05519, 2023.

  • [48]

Y. Wu, Z. Zhang, J. Chen, H. Tang, D. Li, Y. Fang, L. Zhu, E. Xie, H. Yin, L. Yi, et al.

Vila-u: a unified foundation model integrating visual understanding and generation.

arXiv preprint arXiv:2409.04429, 2024b.

  • [49]

Z. Wu, X. Chen, Z. Pan, X. Liu, W. Liu, D. Dai, H. Gao, Y. Ma, C. Wu, B. Wang, et al.

Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.

arXiv preprint arXiv:2412.10302, 2024c.

  • [50]

J. Xie, W. Mao, Z. Bai, D. J. Zhang, W. Wang, K. Q. Lin, Y. Gu, Z. Chen, Z. Yang, and M. Z. Shou.

Show-o: One single transformer to unify multimodal understanding and generation.

arXiv preprint arXiv:2408.12528, 2024.

  • [51]

W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang.

Mm-vet: Evaluating large multimodal models for integrated capabilities.

arXiv preprint arXiv:2308.02490, 2023.

  • [52]

X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al.

Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, 2024.

  • [53]

X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer.

Sigmoid loss for language image pre-training.

In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023.

  • [54]

C. Zhao, Y. Song, W. Wang, H. Feng, E. Ding, Y. Sun, X. Xiao, and J. Wang.

Monoformer: One transformer for both diffusion and autoregression.

arXiv preprint arXiv:2409.16280, 2024.

  • [55]

C. Zhou, L. Yu, A. Babu, K. Tirumala, M. Yasunaga, L. Shamis, J. Kahn, X. Ma, L. Zettlemoyer, and O. Levy.

Transfusion: Predict the next token and diffuse images with one multi-modal model.

arXiv preprint arXiv:2408.11039, 2024.

  • [56]

Y. Zhu, M. Zhu, N. Liu, Z. Ou, X. Mou, and J. Tang.

Llava-phi: Efficient multi-modal assistant with small language model.

arXiv preprint arXiv:2401.02330, 2024.

  • [57]

L. Zhuo, R. Du, H. Xiao, Y. Li, D. Liu, R. Huang, W. Liu, L. Zhao, F.-Y. Wang, Z. Ma, et al.

Lumina-Next: Making Lumina-T2X stronger and faster with Next-DiT.

arXiv preprint arXiv:2406.18583, 2024.