The paper introduces SceMQA, a novel benchmark for scientific multimodal
question answering at the college entrance level. It addresses a critical
educational phase often overlooked in existing benchmarks, spanning high school
to pre-college levels. SceMQA focuses on core science subjects including
Mathematics, Physics, Chemistry, and Biology. It features a blend of
multiple-choice and free-response formats, ensuring a comprehensive evaluation
of AI models' abilities. Additionally, our benchmark provides specific
knowledge points for each problem and detailed explanations for each answer.
SceMQA also uniquely presents problems with identical contexts but varied
questions to facilitate a more thorough and accurate assessment of reasoning
capabilities. In the experiment, we evaluate both open-source and close-source
state-of-the-art Multimodal Large Language Models (MLLMs), across various
experimental settings. The results show that further research and development
are needed in developing more capable MLLM, as highlighted by only 50% to 60%
accuracy achieved by the strongest models. Our benchmark and analysis will be
available at this https URL

本文介绍了 SceMQA，一种用于大学入学阶段的科学多模态问题回答的新型基准。它解决了现有基准中经常被忽视的关键教育阶段，涵盖了高中到预大学的水平。SceMQA 专注于数学、物理、化学和生物等核心科学学科。它采用多项选择和自由回答的形式，确保对人工智能模型能力进行全面评估。此外，我们的基准为每个问题提供了特定的知识点和详细的解释。SceMQA 还通过相同的背景但问题不同的方式，提供了更全面、准确的推理能力评估。在实验中，我们评估了开源和闭源的最先进多模态大语言模型（MLLM），在各种实验设置下进行了评估。结果显示，需要进一步研究和开发更强大的 MLLM，因为最强模型仅达到 50% 到 60% 的准确率。我们的基准和分析将在此 https URL 上提供。