In this paper, we explore a forward-thinking question: Is GPT-4V effective at
low-level data analysis tasks on charts? To this end, we first curate a
large-scale dataset, named ChartInsights, consisting of 89,388 quartets (chart,
task, question, answer) and covering 10 widely-used low-level data analysis
tasks on 7 chart types. Firstly, we conduct systematic evaluations to
understand the capabilities and limitations of 18 advanced MLLMs, which include
12 open-source models and 6 closed-source models. Starting with a standard
textual prompt approach, the average accuracy rate across the 18 MLLMs is
36.17%. Among all the models, GPT-4V achieves the highest accuracy, reaching
56.13%. To understand the limitations of multimodal large models in low-level
data analysis tasks, we have designed various experiments to conduct an
in-depth test of capabilities of GPT-4V. We further investigate how visual
modifications to charts, such as altering visual elements (e.g. changing color
schemes) and introducing perturbations (e.g. adding image noise), affect
performance of GPT-4V. Secondly, we present 12 experimental findings. These
findings suggest potential of GPT-4V to revolutionize interaction with charts
and uncover the gap between human analytic needs and capabilities of GPT-4V.
Thirdly, we propose a novel textual prompt strategy, named Chain-of-Charts,
tailored for low-level analysis tasks, which boosts model performance by
24.36%, resulting in an accuracy of 80.49%. Furthermore, by incorporating a
visual prompt strategy that directs attention of GPT-4V to question-relevant
visual elements, we further improve accuracy to 83.83%. Our study not only
sheds light on the capabilities and limitations of GPT-4V in low-level data
analysis tasks but also offers valuable insights for future research.

我们研究了 GPT-4V 在图表的低级数据分析任务中的有效性，并通过实验结果发现其潜力和局限性，同时提出了一种名为 “Chain-of-Charts” 的新颖文本提示策略，能够提高模型性能并揭示人类分析需求与 GPT-4V 能力之间的差距。