Recent advancements in large vision-language models (LVLMs) have led to
significant progress in generating natural language descriptions for visual
content and thus enhancing various applications. One issue with these powerful
models is that they sometimes produce texts that are factually inconsistent
with the visual input. While there has been some effort to mitigate such
inconsistencies in natural image captioning, the factuality of generated
captions for structured document images, such as charts, has not received as
much scrutiny, posing a potential threat to information reliability in critical
applications. This work delves into the factuality aspect by introducing a
comprehensive typology of factual errors in generated chart captions. A
large-scale human annotation effort provides insight into the error patterns
and frequencies in captions crafted by various chart captioning models,
ultimately forming the foundation of a novel dataset, CHOCOLATE. Our analysis
reveals that even state-of-the-art models, including GPT-4V, frequently produce
captions laced with factual inaccuracies. In response to this challenge, we
establish the new task of Chart Caption Factual Error Correction and introduce
CHARTVE, a model for visual entailment that outperforms proprietary and
open-source LVLMs in evaluating factual consistency. Furthermore, we propose
C2TFEC, an interpretable two-stage framework that excels at correcting factual
errors. This work inaugurates a new domain in factual error correction for
chart captions, presenting a novel evaluation mechanism, and demonstrating an
effective approach to ensuring the factuality of generated chart captions.

通过引入事实性错误的全面分类学，该研究分析各种图表字幕模型生成的错误模式和频率，从而为生成可靠的图表字幕确保事实性的任务奠定了基础。同时，提出了一种有效的两阶段框架来纠正这些错误，以及一种视觉蕴涵模型用于评估事实一致性。