We introduce EXAMS-V, a new challenging multi-discipline multimodal
multilingual exam benchmark for evaluating vision language models. It consists
of 20,932 multiple-choice questions across 20 school disciplines covering
natural science, social science, and other miscellaneous studies, e.g.,
religion, fine arts, business, etc. EXAMS-V includes a variety of multimodal
features such as text, images, tables, figures, diagrams, maps, scientific
symbols, and equations. The questions come in 11 languages from 7 language
families. Unlike existing benchmarks, EXAMS-V is uniquely curated by gathering
school exam questions from various countries, with a variety of education
systems. This distinctive approach calls for intricate reasoning across diverse
languages and relies on region-specific knowledge. Solving the problems in the
dataset requires advanced perception and joint reasoning over the text and the
visual content of the image. Our evaluation results demonstrate that this is a
challenging dataset, which is difficult even for advanced vision-text models
such as GPT-4V and Gemini; this underscores the inherent complexity of the
dataset and its significance as a future benchmark.

我们推出了 EXAMS-V，一个新的多学科多模态多语言考试基准，用于评估视觉语言模型。它包含了 20932 个跨越自然科学、社会科学和其他各种学科的多项选择题，如宗教、美术、商务等。EXAMS-V 包含了多种多模态特征，例如文本、图像、表格、图表、图解、地图、科学符号和方程式。这些问题来自于 7 个语系的 11 种语言。与现有的基准不同，EXAMS-V 是通过收集各个国家的校园考试问题，采用多样化的教育体系进行精心策划的。这种独特的方法需要进行复杂的推理，涉及多种语言和地域特定的知识。解决数据集中的问题需要对文本和图像内容进行高级感知和联合推理。我们的评估结果表明，即使对于像 GPT-4V 和 Gemini 这样的先进视觉 - 文本模型，这也是一个具有挑战性的数据集；这凸显了数据集的内在复杂性以及作为未来基准的重要性。