We introduce a new benchmark, ChartMimic, aimed at assessing the
visually-grounded code generation capabilities of large multimodal models
(LMMs). ChartMimic utilizes information-intensive visual charts and textual
instructions as inputs, requiring LMMs to generate the corresponding code for
chart rendering. ChartMimic includes 1,000 human-curated (figure, instruction,
code) triplets, which represent the authentic chart use cases found in
scientific papers across various domains(e.g., Physics, Computer Science,
Economics, etc). These charts span 18 regular types and 4 advanced types,
diversifying into 191 subcategories. Furthermore, we propose multi-level
evaluation metrics to provide an automatic and thorough assessment of the
output code and the rendered charts. Unlike existing code generation
benchmarks, ChartMimic places emphasis on evaluating LMMs' capacity to
harmonize a blend of cognitive capabilities, encompassing visual understanding,
code generation, and cross-modal reasoning. The evaluation of 3 proprietary
models and 11 open-weight models highlights the substantial challenges posed by
ChartMimic. Even the advanced GPT-4V, Claude-3-opus only achieve an average
score of 73.2 and 53.7, respectively, indicating significant room for
improvement. We anticipate that ChartMimic will inspire the development of
LMMs, advancing the pursuit of artificial general intelligence.

我们介绍了一个新的基准测试，ChartMimic，旨在评估大型多模态模型（LMMs）的视觉基础代码生成能力。ChartMimic 利用信息密集型的可视化图表和文本说明作为输入，要求 LMMs 生成相应的代码以进行图表渲染。ChartMimic 包含了 1,000 个人工策划的（图表，说明，代码）三元组，代表了在各个领域（如物理学，计算机科学，经济学等）的科学论文中找到的真实图表用例。这些图表涵盖了 18 种常规类型和 4 种高级类型，分为 191 个子类别。此外，我们提出了多级评估指标，以对输出代码和渲染后的图表进行自动和全面的评估。与现有的代码生成基准测试不同，ChartMimic 强调评估 LMMs 在视觉理解、代码生成和跨模态推理等多种认知能力的融合能力。对 3 个专有模型和 11 个开源模型的评估突出了 ChartMimic 所带来的巨大挑战。即使是先进的 GPT-4V 和 Claude-3-opus 平均得分分别为 73.2 和 53.7，表明仍有很大的改进空间。我们预计 ChartMimic 将激发 LMMs 的发展，推动人工通用智能的追求。