References

本文为论文《Qwen2.5-VL：面向细粒度感知与智能体的视觉语言模型》「论文原文」板块的对应章节，按 arXiv 官方全文逐字保留，层级与「中文全文翻译」板块一一对应。

  • Agrawal et al. (2024)

Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al.

Pixtral 12b.

arXiv preprint arXiv:2410.07073, 2024.

  • Alayrac et al. (2022)

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.

Flamingo: a visual language model for few-shot learning.

In NeurIPS, 2022.

  • Anthropic (2024a)

Anthropic.

Claude 3.5 sonnet, 2024a.

URL https://www.anthropic.com/news/claude-3-5-sonnet.

  • Anthropic (2024b)

Anthropic.

Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, 2024b.

URL https://www.anthropic.com/news/3-5-models-and-computer-use.

  • Cassano et al. (2023)

Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda.

MultiPL-E: A scalable and polyglot approach to benchmarking neural code generation.

IEEE Trans. Software Eng., 49(7):3675–3691, 2023.

  • Chen et al. (2024a)

Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang.

Allava: Harnessing gpt4v-synthesized data for a lite vision-language model.

arXiv preprint arXiv:2402.11684, 2024a.

  • Chen et al. (2024b)

Jiacheng Chen, Tianhao Liang, Sherman Siu, Zhengqing Wang, Kai Wang, Yubo Wang, Yuansheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, et al.

Mega-bench: Scaling multimodal evaluation to over 500 real-world tasks.

arXiv preprint arXiv:2410.10563, 2024b.

  • Chen et al. (2024c)

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al.

Are we on the right way for evaluating large vision-language models?

arXiv:2403.20330, 2024c.

  • Chen et al. (2021)

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.

Evaluating large language models trained on code.

CoRR, abs/2107.03374, 2021.

  • Chen et al. (2023)

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai.

Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.

arXiv preprint arXiv:2312.14238, 2023.

  • Chen et al. (2024d)

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al.

Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.

arXiv preprint arXiv:2412.05271, 2024d.

  • Cheng et al. (2024)

Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu.

Seeclick: Harnessing gui grounding for advanced visual gui agents.

arXiv preprint arXiv:2401.10935, 2024.

  • Cobbe et al. (2021)

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.

Training verifiers to solve math word problems.

CoRR, abs/2110.14168, 2021.

  • Dauphin et al. (2017)

Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier.

Language modeling with gated convolutional networks.

In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 933–941. PMLR, 2017.

  • Deepmind (2024)

Google Deepmind.

Introducing gemini 2.0: our new ai model for the agentic era, 2024.

URL https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/.

  • DeepSeek-AI et al. (2024)

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, and Wangding Zeng.

Deepseek-v3 technical report.

CoRR, abs/2412.19437, 2024.

doi: 10.48550/ARXIV.2412.19437.

URL https://doi.org/10.48550/arXiv.2412.19437.

  • Deitke et al. (2024)

Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al.

Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.

arXiv preprint arXiv:2409.17146, 2024.

  • Fang et al. (2024)

Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen.

Mmbench-video: A long-form multi-shot benchmark for holistic video understanding.

arXiv preprint arXiv:2406.14515, 2024.

  • Fu et al. (2023)

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al.

Mme: A comprehensive evaluation benchmark for multimodal large language models.

arXiv:2306.13394, 2023.

  • Fu et al. (2024a)

Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al.

Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.

arXiv:2405.21075, 2024a.

  • Fu et al. (2024b)

Ling Fu, Biao Yang, Zhebin Kuang, Jiajun Song, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, Mingxin Huang, Zhang Li, Guozhi Tang, Bin Shan, Chunhui Lin, Qi Liu, Binghong Wu, Hao Feng, Hao Liu, Can Huang, Jingqun Tang, Wei Chen, Lianwen Jin, Yuliang Liu, and Xiang Bai.

Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning, 2024b.

URL https://arxiv.org/abs/2501.00321.

  • Fu et al. (2024c)

Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna.

Blink: Multimodal large language models can see but not perceive.

In European Conference on Computer Vision, pp. 148–166. Springer, 2024c.

  • Gadre et al. (2023)

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al.

Datacomp: In search of the next generation of multimodal datasets.

arXiv:2304.14108, 2023.

  • Gao et al. (2017)

Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia.

Tall: Temporal activity localization via language query.

In Proceedings of the IEEE international conference on computer vision, pp. 5267–5275, 2017.

  • Gema et al. (2024)

Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al.

Are we done with mmlu?

CoRR, abs/2406.04127, 2024.

  • Ghiasi et al. (2021)

Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph.

Simple copy-paste is a strong data augmentation method for instance segmentation.

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2918–2928, 2021.

  • Guan et al. (2023)

Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou.

Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models.

arXiv:2310.14566, 2023.

  • Guo et al. (2024)

Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue.

Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale.

arXiv preprint arXiv:2412.05237, 2024.

  • Hendrycks et al. (2021)

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.

Measuring mathematical problem solving with the MATH dataset.

In NeurIPS Datasets and Benchmarks, 2021.

  • Hu et al. (2025)

Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu.

Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos.

arXiv preprint arXiv:2501.13826, 2025.

  • Kazemzadeh et al. (2014)

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg.

Referitgame: Referring to objects in photographs of natural scenes.

In EMNLP, 2014.

  • Kembhavi et al. (2016)

Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi.

A diagram is worth a dozen images.

In ECCV, 2016.

  • Kirillov et al. (2023)

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.

Segment anything.

In ICCV, 2023.

  • Lee et al. (2024)

Byung-Kwan Lee, Beomchan Park, Chae Won Kim, and Yong Man Ro.

Moai: Mixture of all intelligence for large language and vision models.

In European Conference on Computer Vision, pp. 273–302. Springer, 2024.

  • Li et al. (2023a)

Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu.

Otterhd: A high-resolution multi-modality model.

arXiv:2311.04219, 2023a.

  • Li et al. (2024a)

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al.

Llava-onevision: Easy visual task transfer.

arXiv preprint arXiv:2408.03326, 2024a.

  • Li et al. (2024b)

Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan.

Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension.

arXiv preprint arXiv:2404.16790, 2024b.

  • Li et al. (2024c)

Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li.

Aria: An open multimodal native mixture-of-experts model.

arXiv preprint arXiv:2410.05993, 2024c.

  • Li et al. (2022a)

Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi.

Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.

In ICML, 2022a.

  • Li et al. (2023b)

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.

Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.

arXiv:2301.12597, 2023b.

  • Li et al. (2025a)

Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua.

Screenspot-pro: Gui grounding for professional high-resolution computer use, 2025a.

URL https://likaixin2000.github.io/papers/ScreenSpot_Pro.pdf.

Preprint.

  • Li et al. (2024d)

Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al.

Mvbench: A comprehensive multi-modal video understanding benchmark.

In CVPR, 2024d.

  • Li et al. (2022b)

Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.

Grounded language-image pre-training.

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965–10975, 2022b.

  • Li et al. (2024e)

Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shenglong Ye, Zhenjiang Jin, Guanzhou Chen, Yinan He, Zhangwei Gao, Erfei Cui, et al.

Omnicorpus: An unified multimodal corpus of 10 billion-level images interleaved with text.

arXiv preprint arXiv:2406.08418, 2024e.

  • Li et al. (2024f)

Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva.

On the effects of data scale on computer control agents.

arXiv preprint arXiv:2406.03679, 2024f.

  • Li et al. (2024g)

Yadong Li, Haoze Sun, Mingan Lin, Tianpeng Li, Guosheng Dong, Tao Zhang, Bowen Ding, Wei Song, Zhenglin Cheng, Yuqi Huo, et al.

Baichuan-omni technical report.

arXiv preprint arXiv:2410.08565, 3(7), 2024g.

  • Li et al. (2025b)

Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al.

Baichuan-omni-1.5 technical report.

arXiv preprint arXiv:2501.15368, 2025b.

  • Li et al. (2024h)

Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang.

Uni-moe: Scaling unified multimodal llms with mixture of experts.

arXiv preprint arXiv:2405.11273, 2024h.

  • Li et al. (2023c)

Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai.

Monkey: Image resolution and text label are important things for large multi-modal models.

arXiv:2311.06607, 2023c.

  • Liang et al. (2025)

Yuxuan Liang, Xu Li, Xiaolei Chen, Haotian Chen, Yi Zheng, Chenghang Lai, Bin Li, and Xiangyang Xue.

Global semantic-guided sub-image feature weight allocation in high-resolution large vision-language models.

arXiv preprint arXiv:2501.14276, 2025.

  • Lin et al. (2024)

Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han.

Vila: On pre-training for visual language models.

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26689–26699, 2024.

  • Liu et al. (2023a)

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.

Improved baselines with visual instruction tuning.

arXiv:2310.03744, 2023a.

  • Liu et al. (2023b)

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.

Visual instruction tuning.

arXiv:2304.08485, 2023b.

  • Liu et al. (2023c)

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chun yue Li, Jianwei Yang, Hang Su, Jun-Juan Zhu, and Lei Zhang.

Grounding dino: Marrying dino with grounded pre-training for open-set object detection.

arXiv:2303.05499, 2023c.

  • Liu et al. (2024a)

Yangzhou Liu, Yue Cao, Zhangwei Gao, Weiyun Wang, Zhe Chen, Wenhai Wang, Hao Tian, Lewei Lu, Xizhou Zhu, Tong Lu, et al.

Mminstruct: A high-quality multi-modal instruction tuning dataset with extensive diversity.

Science China Information Sciences, 67(12):1–16, 2024a.

  • Liu et al. (2023d)

Yuan Liu, Haodong Duan, Bo Li Yuanhan Zhang, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin.

Mmbench: Is your multi-modal model an all-around player?

arXiv:2307.06281, 2023d.

  • Liu et al. (2024b)

Yuan Liu, Zhongyin Zhao, Ziyuan Zhuang, Le Tian, Xiao Zhou, and Jie Zhou.

Points: Improving your vision-language model with affordable strategies.

arXiv preprint arXiv:2409.04828, 2024b.

  • Liu et al. (2024c)

Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou.

Tempcompass: Do video llms really understand videos?

arXiv preprint arXiv:2403.00476, 2024c.

  • Liu et al. (2023e)

Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai.

Ocrbench: On the hidden mystery of ocr in large multimodal models.

arXiv:2305.07895, 2023e.

  • Lu et al. (2024)

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.

Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.

In ICLR, 2024.

  • Mangalam et al. (2023)

Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik.

Egoschema: A diagnostic benchmark for very long-form video language understanding.

In NeurIPS, 2023.

  • Mao et al. (2016)

Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy.

Generation and comprehension of unambiguous object descriptions.

In CVPR, 2016.

  • Masry et al. (2022)

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque.

Chartqa: A benchmark for question answering about charts with visual and logical reasoning.

arXiv:2203.10244, 2022.

  • Mathew et al. (2021a)

Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Ernest Valveny, and C.V. Jawahar.

Infographicvqa.

2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2582–2591, 2021a.

  • Mathew et al. (2021b)

Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar.

Docvqa: A dataset for vqa on document images.

In WACV, 2021b.

  • MiniMax et al. (2025)

MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang, Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang, Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang, and Zijia Wu.

Minimax-01: Scaling foundation models with lightning attention, 2025.

URL https://arxiv.org/abs/2501.08313.

  • Openai (2024)

Openai.

Chatml documents, 2024.

URL https://github.com/openai/openai-python/blob/main/chatml.md.

  • OpenAI (2024)

OpenAI.

Hello gpt-4o, 2024.

URL https://openai.com/index/hello-gpt-4o.

  • Ouyang et al. (2024)

Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, Jin Shi, Fan Wu, Pei Chu, Minghao Liu, Zhenxiang Li, Chao Xu, Bo Zhang, Botian Shi, Zhongying Tu, and Conghui He.

Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations, 2024.

URL https://arxiv.org/abs/2412.07626.

  • Paiss et al. (2023)

Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel.

Teaching clip to count to ten.

In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3170–3180, 2023.

  • Patraucean et al. (2024)

Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al.

Perception test: A diagnostic benchmark for multimodal video models.

In NeurIPS, 2024.

  • Peng et al. (2023)

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei.

Kosmos-2: Grounding multimodal large language models to the world.

arXiv:2306.14824, 2023.

  • Rafailov et al. (2023)

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn.

Direct preference optimization: Your language model is secretly a reward model.

In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.

URL http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html.

  • Rawles et al. (2024)

Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al.

Androidworld: A dynamic benchmarking environment for autonomous agents.

arXiv:2405.14573, 2024.

  • Rein et al. (2023)

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman.

GPQA: A graduate-level Google-proof Q&A benchmark.

CoRR, abs/2311.12022, 2023.

  • Ren et al. (2024)

Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, et al.

Grounding dino 1.5: Advance the” edge” of open-set object detection.

arXiv preprint arXiv:2405.10300, 2024.

  • Riquelme et al. (2021)

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby.

Scaling vision with sparse mixture of experts.

Advances in Neural Information Processing Systems, 34:8583–8595, 2021.

  • Singh et al. (2019)

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.

Towards vqa models that can read.

In CVPR, 2019.

  • Su et al. (2024)

Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu.

Roformer: Enhanced Transformer with rotary position embedding.

Neurocomputing, 568:127063, 2024.

  • Tang et al. (2024)

Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, Yanjie Wang, Yuliang Liu, Hao Liu, Xiang Bai, and Can Huang.

Mtvqa: Benchmarking multilingual text-centric visual question answering.

arXiv:2405.11985, 2024.

  • Team et al. (2023)

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.

Gemini: a family of highly capable multimodal models.

arXiv preprint arXiv:2312.11805, 2023.

  • Tong et al. (2024)

Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al.

Cambrian-1: A fully open, vision-centric exploration of multimodal llms.

arXiv preprint arXiv:2406.16860, 2024.

  • Wang et al. (2024a)

Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al.

Muirbench: A comprehensive benchmark for robust multi-image understanding.

arXiv preprint arXiv:2406.09411, 2024a.

  • Wang et al. (2024b)

Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang.

Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration.

arXiv preprint arXiv:2406.01014, 2024b.

  • Wang et al. (2024c)

Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang.

Mobile-agent: Autonomous multi-modal mobile device agent with visual perception.

arXiv preprint arXiv:2401.16158, 2024c.

  • Wang et al. (2024d)

Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li.

Measuring multimodal mathematical reasoning with math-vision dataset.

arXiv:2402.14804, 2024d.

  • Wang et al. (2024e)

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin.

Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.

arXiv:2409.12191, 2024e.

  • Wang et al. (2024f)

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al.

Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.

arXiv preprint arXiv:2409.12191, 2024f.

  • Wang et al. (2024g)

Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al.

Lvbench: An extreme long video understanding benchmark.

arXiv preprint arXiv:2406.08035, 2024g.

  • Wang et al. (2024h)

Weiyun Wang, Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, Zhe Chen, Wenhai Wang, Qingyun Li, Lewei Lu, Xizhou Zhu, et al.

The all-seeing project v2: Towards general relation comprehension of the open world.

arXiv preprint arXiv:2402.19474, 2024h.

  • Wang et al. (2023)

Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al.

Internimage: Exploring large-scale vision foundation models with deformable convolutions.

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14408–14419, 2023.

  • Wang et al. (2024i)

Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al.

Emu3: Next-token prediction is all you need.

arXiv preprint arXiv:2409.18869, 2024i.

  • Wang et al. (2024j)

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen.

MMLU-Pro: A more robust and challenging multi-task language understanding benchmark.

CoRR, abs/2406.01574, 2024j.

  • Wang et al. (2025)

Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji.

Mobile-agent-e: Self-evolving mobile assistant for complex tasks.

arXiv preprint arXiv:2501.11733, 2025.

  • Wang et al. (2024k)

Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen.

Charxiv: Charting gaps in realistic chart understanding in multimodal llms.

arXiv preprint arXiv:2406.18521, 2024k.

  • Wei et al. (2022)

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou.

Chain of thought prompting elicits reasoning in large language models.

CoRR, abs/2201.11903, 2022.

URL https://arxiv.org/abs/2201.11903.

  • White et al. (2024)

Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum.

LiveBench: A challenging, contamination-free LLM benchmark.

CoRR, abs/2406.19314, 2024.

  • Wu et al. (2024a)

Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li.

Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024a.

URL https://arxiv.org/abs/2407.15754.

  • Wu et al. (2024b)

Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al.

Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.

arXiv preprint arXiv:2412.10302, 2024b.

  • X.AI (2024)

X.AI.

Grok-1.5 vision preview.

https://x.ai/blog/grok-1.5v, 2024.

  • Xiao et al. (2023)

Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan.

Florence-2: Advancing a unified representation for a variety of vision tasks (2023).

URL https://arxiv. org/abs/2311.06242, 2023.

  • Xie et al. (2025)

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Jing Hua Toh, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al.

Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.

Advances in Neural Information Processing Systems, 37:52040–52094, 2025.

  • Xu et al. (2024)

Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong.

Aguvis: Unified pure vision agents for autonomous gui interaction.

arXiv preprint arXiv:2412.04454, 2024.

  • Yang et al. (2024a)

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, et al.

Qwen2.5 technical report.

arXiv:2412.15115, 2024a.

  • Yang et al. (2024b)

Zhibo Yang, Jun Tang, Zhaohai Li, Pengfei Wang, Jianqiang Wan, Humen Zhong, Xuejing Liu, Mingkun Yang, Peng Wang, Shuai Bai, LianWen Jin, and Junyang Lin.

Cc-ocr: A comprehensive and challenging ocr benchmark for evaluating large multimodal models in literacy, 2024b.

URL https://arxiv.org/abs/2412.02210.

  • Ye et al. (2024)

Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, et al.

X-vila: Cross-modality alignment for large language model.

arXiv preprint arXiv:2405.19335, 2024.

  • Ye et al. (2023)

Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou.

mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration.

arXiv:2311.04257, 2023.

  • Yu et al. (2024)

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang.

Mm-vet: Evaluating large multimodal models for integrated capabilities.

In ICML, 2024.

  • Yue et al. (2023)

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al.

Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.

arXiv:2311.16502, 2023.

  • Yue et al. (2024)

Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Ming Yin, Botao Yu, Ge Zhang, et al.

Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark.

arXiv preprint arXiv:2409.02813, 2024.

  • Zhang & Sennrich (2019)

Biao Zhang and Rico Sennrich.

Root mean square layer normalization.

In NeurIPS, 2019.

  • Zhang et al. (2024a)

Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, and Yinfei Yang.

Ferret-v2: An improved baseline for referring and grounding with large language models.

arXiv:2404.07973, 2024a.

  • Zhang et al. (2024b)

Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, et al.

Internlm-xcomposer2. 5-omnilive: A comprehensive multimodal system for long-term streaming video and audio interactions.

arXiv preprint arXiv:2412.09596, 2024b.

  • Zhang et al. (2024c)

Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al.

Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?

In European Conference on Computer Vision, pp. 169–186. Springer, 2024c.

  • Zhang et al. (2024d)

Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan.

Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding.

arXiv preprint arXiv:2406.19389, 2024d.

  • Zhang et al. (2024e)

Tianyu Zhang, Suyuchen Wang, Lu Li, Ge Zhang, Perouz Taslakian, Sai Rajeswar, Jie Fu, Bang Liu, and Yoshua Bengio.

Vcr: Visual caption restoration.

arXiv:2406.06462, 2024e.

  • Zhang et al. (2024f)

Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al.

Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?

arXiv preprint arXiv:2408.13257, 2024f.

  • Zhao et al. (2025)

Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, Zhijian Xu, Chengye Wang, Weifeng Pan, Ziyao Shangguan, Xiangru Tang, Zhenwen Liang, Yixin Liu, Chen Zhao, and Arman Cohan.

Mmvu: Measuring expert-level multi-discipline video understanding, 2025.

URL https://arxiv.org/abs/2501.12380.

  • Zhou et al. (2023)

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou.

Instruction-following evaluation for large language models.

CoRR, abs/2311.07911, 2023.

  • Zhou et al. (2024)

Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu.

Mlvu: A comprehensive benchmark for multi-task long video understanding.

arXiv preprint arXiv:2406.04264, 2024.