Although great progress has been made by previous table understanding methods
including recent approaches based on large language models (LLMs), they rely
heavily on the premise that given tables must be converted into a certain text
sequence (such as Markdown or HTML) to serve as model input. However, it is
difficult to access such high-quality textual table representations in some
real-world scenarios, and table images are much more accessible. Therefore, how
to directly understand tables using intuitive visual information is a crucial
and urgent challenge for developing more practical applications. In this paper,
we propose a new problem, multimodal table understanding, where the model needs
to generate correct responses to various table-related requests based on the
given table image. To facilitate both the model training and evaluation, we
construct a large-scale dataset named MMTab, which covers a wide spectrum of
table images, instructions and tasks. On this basis, we develop Table-LLaVA, a
generalist tabular multimodal large language model (MLLM), which significantly
outperforms recent open-source MLLM baselines on 23 benchmarks under held-in
and held-out settings. The code and data is available at this
this https URL

本研究论文提出了一种新的问题，即多模态表格理解，其中模型需要根据给定的表格图像对各种与表格相关的请求生成正确的响应。为了促进模型的训练和评估，构建了一个名为 MMTab 的大规模数据集，该数据集涵盖了广泛的表格图像、指令和任务。在此基础上，研发了 Table-LLaVA，一种通用的表格多模态大型语言模型（MLLM），在 23 个基准测试中，在保留内部和保留外部设置下，其性能明显优于最近的开源 MLLM 基准线。