Large vision-language models (LVLMs), exemplified by GPT-4V, excel across
diverse tasks involving concrete images from natural scenes. However, their
ability to interpret abstract figures, such as geometry shapes and scientific
plots, remains limited due to a scarcity of training datasets in scientific
domains. To fill this gap, we introduce Multimodal ArXiv, consisting of
ArXivCap and ArXivQA, for enhancing LVLMs scientific comprehension. ArXivCap is
a figure-caption dataset comprising 6.4M images and 3.9M captions sourced from
572K ArXiv papers spanning various scientific domains. Drawing from ArXivCap,
we introduce ArXivQA, a question-answering dataset generated by prompting
GPT-4V based on scientific figures. ArXivQA greatly enhances LVLMs'
mathematical reasoning capabilities, achieving a 10.4% absolute accuracy gain
on a multimodal mathematical reasoning benchmark. Furthermore, employing
ArXivCap, we devise four vision-to-text tasks for benchmarking LVLMs.
Evaluation results with state-of-the-art LVLMs underscore their struggle with
the nuanced semantics of academic figures, with domain-specific training
yielding substantial performance gains. Our error analysis uncovers
misinterpretations of visual context, recognition errors, and the production of
overly simplified captions by current LVLMs, shedding light on future
improvements.

通过引入多模态 ArXiv 数据集，包括图像标题数据集 ArXivCap 和基于科学图表产生的问答数据集 ArXivQA，以提高大型视觉 - 语言模型（LVLMs）在科学理解方面的能力，并通过四个视觉到文本任务对 LVLMs 进行了基准评估，结果表明 LVLMs 在学术图表的微妙语义理解方面存在困难，特定领域的训练能够显著提高性能。

提升大型视觉 - 语言模型对科学理解的多模态 ArXiv 数据集

Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of  Large Vision-Language Models

We propose novel evaluations for mathematical reasoning capabilities of Large
Language Models (LLMs) based on mathematical misconceptions. Our primary
approach is to simulate LLMs as a novice learner and an expert tutor, aiming to
identify the incorrect answer to math question resulted from a specific
misconception and to recognize the misconception(s) behind an incorrect answer,
respectively. Contrary to traditional LLMs-based mathematical evaluations that
focus on answering math questions correctly, our approach takes inspirations
from principles in educational learning sciences. We explicitly ask LLMs to
mimic a novice learner by answering questions in a specific incorrect manner
based on incomplete knowledge; and to mimic an expert tutor by identifying
misconception(s) corresponding to an incorrect answer to a question. Using
simple grade-school math problems, our experiments reveal that, while LLMs can
easily answer these questions correctly, they struggle to identify 1) the
incorrect answer corresponding to specific incomplete knowledge
(misconceptions); 2) the misconceptions that explain particular incorrect
answers. Our study indicates new opportunities for enhancing LLMs' math
reasoning capabilities, especially on developing robust student simulation and
expert tutoring models in the educational applications such as intelligent
tutoring systems.

我们提出了一种基于数学误解的大型语言模型（LLM）数学推理能力的新评估方法。我们通过模拟 LLMs 作为初学者和专家导师，旨在识别由于特定误解导致的错误答案，并识别一个错误答案背后的误解。与传统基于 LLMs 的数学评估侧重于正确回答数学问题不同，我们的方法受到教育学习科学原则的启发。我们要求 LLMs 明确地模仿初学者通过基于不完整知识的特定错误方式回答问题，并模仿专家导师识别与问题的错误答案相对应的误解。通过简单的小学数学问题实验，我们发现，尽管 LLMs 可以轻松正确回答这些问题，但它们难以识别：1）与特定不完整知识（误解）相对应的错误答案；2）解释特定错误答案的误解。我们的研究指出了增强 LLMs 数学推理能力的新机会，尤其是在教育应用中开发健壮的学生模拟和专家辅导模型方面。