Chart interpretation is crucial for visual data analysis, but accurately extracting information from charts poses significant challenges for automated models. This study investigates the fine-tuning of DEPLOT, a modality conversion module that translates the image of a plot or chart to a linearized table, on a custom dataset of 50,000 bar charts. The dataset comprises simple, stacked, and grouped bar charts, targeting the unique structural features of these visualizations. The finetuned DEPLOT model is evaluated against its base version using a test set of 1,000 images and two metrics: Relative Mapping Similarity (RMS), which measures categorical mapping accuracy, and Relative Number Set Similarity (RNSS), which evaluates numerical interpretation accuracy. To further explore the reasoning capabilities of large language models (LLMs), we curate an additional set of 100 bar chart images paired with question answer sets. Our findings demonstrate that providing a structured intermediate table alongside the image significantly enhances LLM reasoning performance compared to direct image queries.

本研究解决了从图表中准确提取信息的困难，特别是在视觉数据分析中。通过对DEPLOT模块进行微调，使其能将图表转换为线性化表格，从而有效提高了大型语言模型的推理能力。结果显示，提供结构化的中间表格能显著提升模型在处理图表时的推理表现。

利用中间结构表示增强金融视觉问答在视觉语言模型中的应用