Charts play a vital role in data visualization, understanding data patterns,
and informed decision-making. However, their unique combination of graphical
elements (e.g., bars, lines) and textual components (e.g., labels, legends)
poses challenges for general-purpose multimodal models. While vision-language
models trained on chart data excel in comprehension, they struggle with
generalization and require task-specific fine-tuning. To address these
challenges, we propose ChartAssistant, a chart-based vision-language model for
universal chart comprehension and reasoning. ChartAssistant leverages ChartSFT,
a comprehensive dataset covering diverse chart-related tasks with basic and
specialized chart types. It undergoes a two-stage training process, starting
with pre-training on chart-to-table parsing to align chart and text, followed
by multitask instruction-following fine-tuning. This approach enables
ChartAssistant to achieve competitive performance across various chart tasks
without task-specific fine-tuning. Experimental results demonstrate significant
performance gains over the state-of-the-art UniChart method, outperforming
OpenAI's GPT-4V(ision) on real-world chart data. The code and data are
available at this https URL

通过 ChartAssistant 这一基于图表的视觉语言模型，本研究提出了一种解决图表理解的通用模型，经实验证明在各种图表任务上取得了显著性能提升。

ChartAssisstant：通过图表 - 表格预训练和多任务指导调优的通用图表多模态语言模型

ChartAssisstant: A Universal Chart Multimodal Language Model via  Chart-to-Table Pre-training and Multitask Instruction Tuning

Multimodal Large Language Models (MLLMs) have demonstrated remarkable
multimodal understanding and generation capabilities. However, their
understanding of synthetic charts is limited, while existing benchmarks are
simplistic and the charts deviate significantly from real-world examples,
making it challenging to accurately assess MLLMs' chart comprehension
abilities. Hence, a challenging benchmark is essential for investigating
progress and uncovering the limitations of current MLLMs on chart data. In this
work, we propose to examine chart comprehension through more complex visual
logic and introduce ChartBench, a comprehensive chart benchmark to accurately
measure MLLMs' fundamental chart comprehension and data reliability.
Specifically, ChartBench consists of \textbf{41} categories, \textbf{2K}
charts, and \textbf{16K} QA annotations. While significantly expanding chart
types, ChartBench avoids direct labelling of data points, which requires MLLMs
to infer values akin to humans by leveraging elements like color, legends, and
coordinate systems. We also introduce an improved metric, \textit{Acc+}, which
accurately reflects MLLMs' chart comprehension abilities while avoiding
labor-intensive manual evaluations or costly GPT-based evaluations. We conduct
evaluations on \textbf{12} mainstream open-source models and \textbf{2}
outstanding proprietary models. Through extensive experiments, we reveal the
limitations of MLLMs on charts and provide insights to inspire the community to
pay closer attention to MLLMs' chart comprehension abilities. The benchmark and
code will be publicly available for research.

通过更复杂的视觉逻辑和引入 ChartBench 基准，准确衡量 MLLMs 在图表数据中的基本理解能力和数据可靠性，并揭示了它们的局限性，以激发研究界对 MLLMs 的图表理解能力更加关注。

ChartBench：图表中的复杂可视推理基准

ChartBench: A Benchmark for Complex Visual Reasoning in Charts

Multi-modal large language models have demonstrated impressive performances
on most vision-language tasks. However, the model generally lacks the
understanding capabilities for specific domain data, particularly when it comes
to interpreting chart figures. This is mainly due to the lack of relevant
multi-modal instruction tuning datasets. In this article, we create a
high-quality instruction-tuning dataset leveraging GPT-4. We develop a
multi-step data generation process in which different steps are responsible for
generating tabular data, creating chart figures, and designing instruction
tuning data separately. Our method's flexibility enables us to generate
diverse, high-quality instruction-tuning data consistently and efficiently
while maintaining a low resource expenditure. Additionally, it allows us to
incorporate a wider variety of chart and task types not yet featured in
existing datasets. Next, we introduce ChartLlama, a multi-modal large language
model that we've trained using our created dataset. ChartLlama outperforms all
prior methods in ChartQA, Chart-to-text, and Chart-extraction evaluation
benchmarks. Additionally, ChartLlama significantly improves upon the baseline
in our specially compiled chart dataset, which includes new chart and task
types. The results of ChartLlama confirm the value and huge potential of our
proposed data generation method in enhancing chart comprehension.

通过创建高质量的指令调整数据集，并使用这个数据集训练多模态大型语言模型 ChartLlama，本研究提出的数据生成方法可以有效地提高图表理解能力，并在 ChartQA、图表转文本和图表提取等评估中明显超越以往的方法，证实了其巨大潜力。