While large multi-modal models (LMM) have shown notable progress in
multi-modal tasks, their capabilities in tasks involving dense textual content
remains to be fully explored. Dense text, which carries important information,
is often found in documents, tables, and product descriptions. Understanding
dense text enables us to obtain more accurate information, assisting in making
better decisions. To further explore the capabilities of LMM in complex text
tasks, we propose the DT-VQA dataset, with 170k question-answer pairs. In this
paper, we conduct a comprehensive evaluation of GPT4V, Gemini, and various
open-source LMMs on our dataset, revealing their strengths and weaknesses.
Furthermore, we evaluate the effectiveness of two strategies for LMM: prompt
engineering and downstream fine-tuning. We find that even with automatically
labeled training datasets, significant improvements in model performance can be
achieved. We hope that this research will promote the study of LMM in dense
text tasks. Code will be released at
this https URL

本研究通过在 DT-VQA 数据集上对 GPT4V、Gemini 和不同的开源 LMM 进行全面评估，揭示了它们的优点和缺点，并评估了 LMM 的两种策略：prompt engineering 和 downstream fine-tuning。研究发现，即使使用自动标记的训练数据集，模型性能也能显著提高，希望本研究能促进 LMM 在密集文本任务中的研究。