In this paper, we establish a benchmark for table visual question answering,
referred to as the TableVQA-Bench, derived from pre-existing table
question-answering (QA) and table structure recognition datasets. It is
important to note that existing datasets have not incorporated images or QA
pairs, which are two crucial components of TableVQA. As such, the primary
objective of this paper is to obtain these necessary components. Specifically,
images are sourced either through the application of a \textit{stylesheet} or
by employing the proposed table rendering system. QA pairs are generated by
exploiting the large language model (LLM) where the input is a text-formatted
table. Ultimately, the completed TableVQA-Bench comprises 1,500 QA pairs. We
comprehensively compare the performance of various multi-modal large language
models (MLLMs) on TableVQA-Bench. GPT-4V achieves the highest accuracy among
commercial and open-sourced MLLMs from our experiments. Moreover, we discover
that the number of vision queries plays a significant role in TableVQA
performance. To further analyze the capabilities of MLLMs in comparison to
their LLM backbones, we investigate by presenting image-formatted tables to
MLLMs and text-formatted tables to LLMs, respectively. Our findings suggest
that processing visual inputs is more challenging than text inputs, as
evidenced by the lower performance of MLLMs, despite generally requiring higher
computational costs than LLMs. The proposed TableVQA-Bench and evaluation codes
are available at
\href{https://github.com/naver-ai/tablevqabench}{this https URL}.

本文介绍了一个名为 TableVQA-Bench 的基准，用于表格视觉问答，该基准由现有的表格问答和表格结构识别数据集派生而来。通过使用样式表或提出的表格渲染系统，获取图像，并通过利用大型语言模型生成 QA 问题。我们在 TableVQA-Bench 上全面比较了不同的多模态大型语言模型的性能，其中 GPT-4V 在商业和开源的多模态大型语言模型中表现最高的准确率。研究结果表明，对于 TableVQA 而言，视觉输入的处理比文本输入更具挑战性。