References

本文为论文《Gemma 3：轻量开放的多模态模型家族》「中文全文翻译」板块的对应章节，与「论文原文」板块同一章节一一对应，层级保持一致。

  • (1) Realworldqa.

https://x.ai/news/grok-1.5v.

  • Acharya et al. (2018)

M. Acharya, K. Kafle, and C. Kanan.

Tallyqa: Answering complex counting questions.

In AAAI, 2018.

  • Agarwal et al. (2024)

R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. R. Garea, M. Geist, and O. Bachem.

On-policy distillation of language models: Learning from self-generated mistakes.

In ICLR, 2024.

  • Ainslie et al. (2023)

J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai.

Gqa: Training generalized multi-query transformer models from multi-head checkpoints.

arXiv 预印本 arXiv:2305.13245, 2023.

  • Anil et al. (2018)

R. Anil, G. Pereyra, A. Passos, R. Ormandi, G. E. Dahl, and G. E. Hinton.

Large scale distributed neural network training through online distillation.

arXiv 预印本 arXiv:1804.03235, 2018.

  • Anil et al. (2023)

R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al.

Palm 2 technical report.

arXiv 预印本 arXiv:2305.10403, 2023.

  • Artetxe et al. (2020)

M. Artetxe, S. Ruder, and D. Yogatama.

On the cross-lingual transferability of monolingual representations.

In ACL, 2020.

  • Asai et al. (2020)

A. Asai, J. Kasai, J. H. Clark, K. Lee, E. Choi, and H. Hajishirzi.

Xor qa: Cross-lingual open-retrieval question answering.

arXiv 预印本 arXiv:2010.11856, 2020.

  • Austin et al. (2021)

J. Austin, A. Odena, M. I. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. J. Cai, M. Terry, Q. V. Le, and C. Sutton.

Program synthesis with large language models.

CoRR, abs/2108.07732, 2021.

  • Barham et al. (2022)

P. Barham, A. Chowdhery, J. Dean, S. Ghemawat, S. Hand, D. Hurt, M. Isard, H. Lim, R. Pang, S. Roy, B. Saeta, P. Schuh, R. Sepassi, L. E. Shafey, C. A. Thekkath, and Y. Wu.

Pathways: Asynchronous distributed dataflow for ml, 2022.

  • Beltagy et al. (2020)

I. Beltagy, M. E. Peters, and A. Cohan.

Longformer: The long-document transformer.

arXiv 预印本 arXiv:2004.05150, 2020.

  • Biderman et al. (2023)

S. Biderman, U. Prashanth, L. Sutawika, H. Schoelkopf, Q. Anthony, S. Purohit, and E. Raff.

Emergent and predictable memorization in large language models.

NeurIPS, 36:28072–28090, 2023.

  • Bisk et al. (2019)

Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi.

PIQA: reasoning about physical commonsense in natural language.

CoRR, abs/1911.11641, 2019.

  • Carlini et al. (2021)

N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al.

Extracting training data from large language models.

In USENIX, 2021.

  • Carlini et al. (2022)

N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang.

Quantifying memorization across neural language models.

arXiv 预印本 arXiv:2202.07646, 2022.

  • Chameleon Team (2024)

Chameleon Team.

Chameleon: Mixed-modal early-fusion foundation models.

arXiv 预印本 arXiv:2405.09818, 2024.

  • Chen et al. (2021)

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba.

Evaluating large language models trained on code.

CoRR, abs/2107.03374, 2021.

  • Chen et al. (2023)

S. Chen, S. Wong, L. Chen, and Y. Tian.

Extending context window of large language models via positional interpolation.

arXiv 预印本 arXiv:2306.15595, 2023.

  • Chen et al. (2015)

X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick.

Microsoft coco captions: Data collection and evaluation server.

ArXiv, abs/1504.00325, 2015.

  • Chiang et al. (2024)

W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica.

Chatbot arena: An open platform for evaluating llms by human preference, 2024.

  • Chollet (2019)

F. Chollet.

On the measure of intelligence.

arXiv 预印本 arXiv:1911.01547, 2019.

  • Chowdhery et al. (2022)

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel.

Palm: Scaling language modeling with pathways, 2022.

  • Chung et al. (2023)

H. W. Chung, N. Constant, X. Garcia, A. Roberts, Y. Tay, S. Narang, and O. Firat.

Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining, 2023.

  • Clark et al. (2019)

C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova.

Boolq: Exploring the surprising difficulty of natural yes/no questions.

CoRR, abs/1905.10044, 2019.

  • Cobbe et al. (2021)

K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman.

Training verifiers to solve math word problems.

CoRR, abs/2110.14168, 2021.

  • DeepSeek-AI (2025)

DeepSeek-AI.

Deepseek-r1: Incentivizing reasoningt learning, 2025.

  • Dehghani et al. (2023)

M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, et al.

Scaling vision transformers to 22 billion parameters.

In ICML, 2023.

  • Deutsch et al. (2025)

D. Deutsch, E. Briakou, I. Caswell, M. Finkelstein, R. Galor, J. Juraska, G. Kovacs, A. Lui, R. Rei, J. Riesa, S. Rijhwani, P. Riley, E. Salesky, F. Trabelsi, S. Winkler, B. Zhang, and M. Freitag.

Wmt24++: Expanding the language coverage of wmt24 to 55 languages & dialects, 2025.

  • Dosovitskiy (2020)

A. Dosovitskiy.

An image is worth 16x16 words: Transformers for image recognition at scale.

arXiv 预印本 arXiv:2010.11929, 2020.

  • Dua et al. (2019)

D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner.

DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.

In ACL, 2019.

  • Fatemi et al. (2024)

B. Fatemi, M. Kazemi, A. Tsitsulin, K. Malkan, J. Yim, J. Palowitch, S. Seo, J. Halcrow, and B. Perozzi.

Test of time: A benchmark for evaluating llms on temporal reasoning.

arXiv 预印本 arXiv:2406.09170, 2024.

  • Fu et al. (2024)

X. Fu, Y. Hu, B. Li, Y. Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W.-C. Ma, and R. Krishna.

Blink: Multimodal large language models can see but not perceive.

ArXiv, abs/2404.12390, 2024.

  • Gehring et al. (2024)

J. Gehring, K. Zheng, J. Copet, V. Mella, T. Cohen, and G. Synnaeve.

Rlef: Grounding code llms in execution feedback with reinforcement learning.

arXiv 预印本 arXiv:2410.02089, 2024.

  • Gemini Team (2023)

Gemini Team.

Gemini: A family of highly capable multimodal models, 2023.

  • Gemini Team (2024)

Gemini Team.

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.

  • Gemma Team (2024a)

Gemma Team.

Gemma: Open models based on gemini research and technology, 2024a.

  • Gemma Team (2024b)

Gemma Team.

Gemma 2: Improving open language models at a practical size.

arXiv 预印本 arXiv:2408.00118, 2024b.

  • Goldman et al. (2025)

O. Goldman, U. Shaham, D. Malkin, S. Eiger, A. Hassidim, Y. Matias, J. Maynez, A. M. Gilady, J. Riesa, S. Rijhwani, L. Rimell, I. Szpektor, R. Tsarfaty, and M. Eyal.

Eclektic: a novel challenge set for evaluation of cross-lingual knowledge transfer, 2025.

  • Goyal et al. (2022)

N. Goyal, C. Gao, V. Chaudhary, P.-J. Chen, G. Wenzek, D. Ju, S. Krishnan, M. Ranzato, F. Guzmán, and A. Fan.

The flores-101 evaluation benchmark for low-resource and multilingual machine translation.

ACL, 2022.

  • Goyal et al. (2017)

Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh.

Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.

In CVPR, 2017.

  • Hendrycks et al. (2020)

D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt.

Measuring massive multitask language understanding.

CoRR, abs/2009.03300, 2020.

  • Hendrycks et al. (2021)

D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.

Measuring mathematical problem solving with the math dataset.

NeurIPS, 2021.

  • Hessel et al. (2022)

J. Hessel, A. Marasović, J. D. Hwang, L. Lee, J. Da, R. Zellers, R. Mankoff, and Y. Choi.

Do androids laugh at electric sheep? humor" understanding" benchmarks from the new yorker caption contest.

arXiv 预印本 arXiv:2209.06293, 2022.

  • Hinton et al. (2015)

G. Hinton, O. Vinyals, and J. Dean.

Distilling the knowledge in a neural network.

arXiv 预印本 arXiv:1503.02531, 2015.

  • Hsieh et al. (2024)

C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg.

Ruler: What’s the real context size of your long-context language models?

arXiv 预印本 arXiv:2404.06654, 2024.

  • Ippolito et al. (2022)

D. Ippolito, F. Tramèr, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette-Choo, and N. Carlini.

Preventing verbatim memorization in language models gives a false sense of privacy.

arXiv 预印本 arXiv:2210.17546, 2022.

  • Jacob et al. (2018)

B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko.

Quantization and training of neural networks for efficient integer-arithmetic-only inference.

In CVPR, 2018.

  • Joshi et al. (2017)

M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer.

Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.

CoRR, abs/1705.03551, 2017.

  • Kazemi et al. (2023)

M. Kazemi, H. Alvari, A. Anand, J. Wu, X. Chen, and R. Soricut.

Geomverse: A systematic evaluation of large models for geometric reasoning.

arXiv 预印本 arXiv:2312.12241, 2023.

  • Kazemi et al. (2024a)

M. Kazemi, N. Dikkala, A. Anand, P. Dević, I. Dasgupta, F. Liu, B. Fatemi, P. Awasthi, D. Guo, S. Gollapudi, and A. Qureshi.

Remi: A dataset for reasoning with multiple images.

ArXiv, abs/2406.09175, 2024a.

  • Kazemi et al. (2024b)

M. Kazemi, Q. Yuan, D. Bhatia, N. Kim, X. Xu, V. Imbrasaite, and D. Ramachandran.

Boardgameqa: A dataset for natural language reasoning with contradictory information.

NeurIPS, 36, 2024b.

  • Kazemi et al. (2025)

M. Kazemi, B. Fatemi, H. Bansal, J. Palowitch, C. Anastasiou, S. V. Mehta, L. K. Jain, V. Aglietti, D. Jindal, P. Chen, et al.

Big-bench extra hard.

arXiv 预印本 arXiv:2502.19187, 2025.

  • Kembhavi et al. (2016)

A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi.

A diagram is worth a dozen images.

ArXiv, abs/1603.07396, 2016.

  • Kıcıman et al. (2023)

E. Kıcıman, R. Ness, A. Sharma, and C. Tan.

Causal reasoning and large language models: Opening a new frontier for causality.

arXiv 预印本 arXiv:2305.00050, 2023.

  • Kudo and Richardson (2018)

T. Kudo and J. Richardson.

SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.

2018.

  • Kwiatkowski et al. (2019)

T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov.

Natural questions: A benchmark for question answering research.

ACL, 2019.

  • Lambert et al. (2024)

N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al.

T\backslash" ulu 3: Pushing frontiers in open language model post-training.

arXiv 预印本 arXiv:2411.15124, 2024.

  • Lin et al. (2024)

Z. Lin, J. Cui, X. Liao, and X. Wang.

Malla: Demystifying real-world large language model integrated malicious services, 2024.

  • Liu et al. (2024)

H. Liu, C. Li, Q. Wu, and Y. J. Lee.

Visual instruction tuning.

NeurIPS, 36, 2024.

  • LLaMa Team (2024)

LLaMa Team.

The llama 3 herd of models.

arXiv 预印本 arXiv:2407.21783, 2024.

  • Luong et al. (2015)

M. Luong, H. Pham, and C. D. Manning.

Effective approaches to attention-based neural machine translation.

2015.

  • (62)

Macknight, Aung, and Gomes.

Personal Communication.

  • Marino et al. (2019)

K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi.

Ok-vqa: A visual question answering benchmark requiring external knowledge.

In CVPR, 2019.

  • Masry et al. (2022)

A. Masry, X. L. Do, J. Q. Tan, S. Joty, and E. Hoque.

ChartQA: A benchmark for question answering about charts with visual and logical reasoning.

ACL, 2022.

  • Mathew et al. (2020)

M. Mathew, D. Karatzas, R. Manmatha, and C. V. Jawahar.

Docvqa: A dataset for vqa on document images.

WACV, 2020.

  • Mathew et al. (2022)

M. Mathew, V. Bagal, R. Tito, D. Karatzas, E. Valveny, and C. Jawahar.

Infographicvqa.

In WACV, 2022.

  • Mirzadeh et al. (2024)

I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar.

Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models.

arXiv 预印本 arXiv:2410.05229, 2024.

  • Nasr et al. (2023)

M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee.

Scalable extraction of training data from (production) language models.

arXiv 预印本 arXiv:2311.17035, 2023.

  • Nie et al. (2024)

A. Nie, Y. Zhang, A. S. Amdekar, C. Piech, T. B. Hashimoto, and T. Gerstenberg.

Moca: Measuring human-language model alignment on causal and moral judgment tasks.

NeurIPS, 36, 2024.

  • Paiss et al. (2023)

R. Paiss, A. Ephrat, O. Tov, S. Zada, I. Mosseri, M. Irani, and T. Dekel.

Teaching clip to count to ten.

ICCV, 2023.

  • Phuong et al. (2024)

M. Phuong, M. Aitchison, E. Catt, S. Cogan, A. Kaskasoli, V. Krakovna, D. Lindner, M. Rahtz, Y. Assael, S. Hodkinson, H. Howard, T. Lieberum, R. Kumar, M. A. Raad, A. Webson, L. Ho, S. Lin, S. Farquhar, M. Hutter, G. Deletang, A. Ruoss, S. El-Sayed, S. Brown, A. Dragan, R. Shah, A. Dafoe, and T. Shevlane.

Evaluating frontier models for dangerous capabilities, 2024.

  • Radford et al. (2021)

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.

Learning transferable visual models from natural language supervision.

In ICML, pages 8748–8763. PMLR, 2021.

  • Ramé et al. (2024a)

A. Ramé, J. Ferret, N. Vieillard, R. Dadashi, L. Hussenot, P.-L. Cedoz, P. G. Sessa, S. Girgin, A. Douillard, and O. Bachem.

WARP: On the benefits of weight averaged rewarded policies, 2024a.

  • Ramé et al. (2024b)

A. Ramé, N. Vieillard, L. Hussenot, R. Dadashi, G. Cideron, O. Bachem, and J. Ferret.

WARM: On the benefits of weight averaged reward models.

In ICML, 2024b.

  • Rein et al. (2023)

D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman.

Gpqa: A graduate-level google-proof q&a benchmark.

ArXiv, abs/2311.12022, 2023.

  • Ren et al. (2021)

J. Ren, S. Rajbhandari, R. Y. Aminabadi, O. Ruwase, S. Yang, M. Zhang, D. Li, and Y. He.

Zero-offload: Democratizing billion-scale model training.

In USENIX, 2021.

  • Roberts et al. (2023)

A. Roberts, H. W. Chung, G. Mishra, A. Levskaya, J. Bradbury, D. Andor, S. Narang, B. Lester, C. Gaffney, A. Mohiuddin, et al.

Scaling up models and data with t5x and seqio.

JMLR, 2023.

  • Sachdeva et al. (2024)

N. Sachdeva, B. Coleman, W.-C. Kang, J. Ni, L. Hong, E. H. Chi, J. Caverlee, J. McAuley, and D. Z. Cheng.

How to train data-efficient llms.

arXiv 预印本 arXiv:2402.09668, 2024.

  • Sakaguchi et al. (2019)

K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi.

WINOGRANDE: an adversarial winograd schema challenge at scale.

CoRR, abs/1907.10641, 2019.

  • Sánchez et al. (2024)

E. Sánchez, B. Alastruey, C. Ropers, P. Stenetorp, M. Artetxe, and M. R. Costa-jussà.

Linguini: A benchmark for language-agnostic linguistic reasoning.

arXiv 预印本 arXiv:2409.12126, 2024.

  • Sap et al. (2019)

M. Sap, H. Rashkin, D. Chen, R. L. Bras, and Y. Choi.

Socialiqa: Commonsense reasoning about social interactions.

CoRR, abs/1904.09728, 2019.

  • Sessa et al. (2024)

P. G. Sessa, R. Dadashi, L. Hussenot, J. Ferret, N. Vieillard, A. Ramé, B. Shariari, S. Perrin, A. Friesen, G. Cideron, S. Girgin, P. Stanczyk, A. Michi, D. Sinopalnikov, S. Ramos, A. Héliou, A. Severyn, M. Hoffman, N. Momchev, and O. Bachem.

Bond: Aligning llms with best-of-n distillation, 2024.

  • Shah et al. (2024)

K. Shah, N. Dikkala, X. Wang, and R. Panigrahy.

Causal language modeling can elicit search and reasoning capabilities on logic puzzles.

arXiv 预印本 arXiv:2409.10502, 2024.

  • Shevlane et al. (2023)

T. Shevlane, S. Farquhar, B. Garfinkel, M. Phuong, J. Whittlestone, J. Leung, D. Kokotajlo, N. Marchal, M. Anderljung, N. Kolt, L. Ho, D. Siddarth, S. Avin, W. Hawkins, B. Kim, I. Gabriel, V. Bolina, J. Clark, Y. Bengio, P. Christiano, and A. Dafoe.

Model evaluation for extreme risks, 2023.

  • Shi et al. (2023)

F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, D. Das, and J. Wei.

Language models are multilingual chain-of-thought reasoners.

In ICLR, 2023.

  • Singh et al. (2019)

A. Singh, V. Natarjan, M. Shah, Y. Jiang, X. Chen, D. Parikh, and M. Rohrbach.

Towards vqa models that can read.

In CVPR, 2019.

  • Singh et al. (2024a)

H. Singh, N. Gupta, S. Bharadwaj, D. Tewari, and P. Talukdar.

Indicgenbench: a multilingual benchmark to evaluate generation capabilities of llms on indic languages.

arXiv 预印本 arXiv:2404.16816, 2024a.

  • Singh et al. (2024b)

S. Singh, A. Romanou, C. Fourrier, D. I. Adelani, J. G. Ngui, D. Vila-Suero, P. Limkonchotiwat, K. Marchisio, W. Q. Leong, Y. Susanto, R. Ng, S. Longpre, W.-Y. Ko, M. Smith, A. Bosselut, A. Oh, A. F. T. Martins, L. Choshen, D. Ippolito, E. Ferrante, M. Fadaee, B. Ermis, and S. Hooker.

Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation, 2024b.

  • Steiner et al. (2024)

A. Steiner, A. S. Pinto, M. Tschannen, D. Keysers, X. Wang, Y. Bitton, A. Gritsenko, M. Minderer, A. Sherbondy, S. Long, S. Qin, R. Ingle, E. Bugliarello, S. Kazemzadeh, T. Mesnard, I. Alabdulmohsin, L. Beyer, and X. Zhai.

PaliGemma 2: A Family of Versatile VLMs for Transfer.

arXiv 预印本 arXiv:2412.03555, 2024.

  • Suzgun et al. (2022)

M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, and J. Wei.

Challenging big-bench tasks and whether chain-of-thought can solve them, 2022.

  • Tyen et al. (2023)

G. Tyen, H. Mansoor, P. Chen, T. Mak, and V. Cărbune.

Llms cannot find reasoning errors, but can correct them!

arXiv 预印本 arXiv:2311.08516, 2023.

  • Vaswani et al. (2017)

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.

Attention is all you need.

2017.

  • Vodrahalli et al. (2024)

K. Vodrahalli, S. Ontanon, N. Tripuraneni, K. Xu, S. Jain, R. Shivanna, J. Hui, N. Dikkala, M. Kazemi, B. Fatemi, et al.

Michelangelo: Long context evaluations beyond haystacks via latent structure queries.

arXiv 预印本 arXiv:2409.12640, 2024.

  • Wang et al. (2024)

Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al.

Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.

In NeurIPS, 2024.

  • Weidinger et al. (2021)

L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, Z. Kenton, S. Brown, W. Hawkins, T. Stepleton, C. Biles, A. Birhane, J. Haas, L. Rimell, L. A. Hendricks, W. Isaac, S. Legassick, G. Irving, and I. Gabriel.

Ethical and social risks of harm from language models, 2021.

  • White et al. (2024)

C. White, S. Dooley, M. Roberts, A. Pal, B. Feuer, S. Jain, R. Shwartz-Ziv, N. Jain, K. Saifullah, S. Naidu, et al.

Livebench: A challenging, contamination-free llm benchmark.

arXiv 预印本 arXiv:2406.19314, 2024.

  • Wortsman et al. (2023)

M. Wortsman, P. J. Liu, L. Xiao, K. Everett, A. Alemi, B. Adlam, J. D. Co-Reyes, I. Gur, A. Kumar, R. Novak, et al.

Small-scale proxies for large-scale transformer training instabilities.

arXiv 预印本 arXiv:2309.14322, 2023.

  • XLA (2019)

XLA.

Xla: Optimizing compiler for tensorflow, 2019.

URL https://www.tensorflow.org/xla.

  • Xu et al. (2021)

Y. Xu, H. Lee, D. Chen, B. A. Hechtman, Y. Huang, R. Joshi, M. Krikun, D. Lepikhin, A. Ly, M. Maggioni, R. Pang, N. Shazeer, S. Wang, T. Wang, Y. Wu, and Z. Chen.

GSPMD: general and scalable parallelization for ML computation graphs.

2021.

  • Yamada et al. (2023)

Y. Yamada, Y. Bao, A. K. Lampinen, J. Kasai, and I. Yildirim.

Evaluating spatial understanding of large language models.

arXiv 预印本 arXiv:2310.14540, 2023.

  • Yang et al. (2019)

K. Yang, O. Russakovsky, and J. Deng.

Spatialsense: An adversarially crowdsourced benchmark for spatial relation recognition.

ICCV, 2019.

  • Yue et al. (2023)

X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen.

Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.

CVPR, 2023.

  • Zellers et al. (2019)

R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi.

HellaSwag: Can a machine really finish your sentence?

In ACL, 2019.

  • Zhai et al. (2023)

X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer.

Sigmoid loss for language image pre-training.

In CVPR, 2023.

  • Zhang and Sennrich (2019)

B. Zhang and R. Sennrich.

Root mean square layer normalization.

2019.

  • Zhang et al. (2024)

J. Zhang, L. Jain, Y. Guo, J. Chen, K. L. Zhou, S. Suresh, A. Wagenmaker, S. Sievert, T. Rogers, K. Jamieson, et al.

Humor in ai: Massive scale crowd-sourced preferences and benchmarks for cartoon captioning.

arXiv 预印本 arXiv:2406.10522, 2024.

  • Zhong et al. (2023)

W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan.

Agieval: A human-centric benchmark for evaluating foundation models, 2023.