Building cross-model intelligence that can understand charts and communicate
the salient information hidden behind them is an appealing challenge in the
vision and language(V+L) community. The capability to uncover the underlined
table data of chart figures is a critical key to automatic chart understanding.
We introduce ChartT5, a V+L model that learns how to interpret table
information from chart images via cross-modal pre-training on plot table pairs.
Specifically, we propose two novel pre-training objectives: Masked Header
Prediction (MHP) and Masked Value Prediction (MVP) to facilitate the model with
different skills to interpret the table information. We have conducted
extensive experiments on chart question answering and chart summarization to
verify the effectiveness of the proposed pre-training strategies. In
particular, on the ChartQA benchmark, our ChartT5 outperforms the
state-of-the-art non-pretraining methods by over 8% performance gains.

介绍了一种名为 ChartT5 的 V+L 模型，通过对绘图表格对进行跨模态预训练学习，具备了解释图表信息的能力，其采用的两种新颖预训练目标 Masked Header Prediction (MHP) 和 Masked Value Prediction (MVP) 有效提高了模型的表格信息解释能力，在答题和总结图表方面的表现都超过了同类方法 8% 以上。