In this work, we present the MM-MATH dataset, a novel benchmark developed to
rigorously evaluate the performance of advanced large language and multimodal
models - including but not limited to GPT-4, GPT-4V, and Claude - within the
domain of geometric computation. This dataset comprises 5,929 meticulously
crafted geometric problems, each paired with a corresponding image, aimed at
mirroring the complexity and requirements typical of ninth-grade mathematics.
The motivation behind MM-MATH stems from the burgeoning interest and
significant strides in multimodal technology, which necessitates a paradigm
shift in assessment methodologies from mere outcome analysis to a more holistic
evaluation encompassing reasoning and procedural correctness. Despite
impressive gains in various benchmark performances, our analysis uncovers a
persistent and notable deficiency in these models' ability to parse and
interpret geometric information accurately from images, accounting for over 60%
of observed errors. By deploying a dual-focused evaluation approach, examining
both the end results and the underlying problem-solving processes, we unearthed
a marked discrepancy between the capabilities of current multimodal models and
human-level proficiency. The introduction of MM-MATH represents a tripartite
contribution to the field: it not only serves as a comprehensive and
challenging benchmark for assessing geometric problem-solving prowess but also
illuminates critical gaps in textual and visual comprehension that current
models exhibit. Through this endeavor, we aspire to catalyze further research
and development aimed at bridging these gaps, thereby advancing the state of
multimodal model capabilities to new heights.

通过 MM-MATH 数据集，该研究旨在评估多模态模型在几何计算领域的性能，发现当前模型从图像中解析和解释几何信息存在显著不足，强调评估方法应包括推理和过程正确性，以填补文本和图像理解方面的关键差距，以此激发进一步研究和发展，推动多模态模型能力的提升。