We study the problem of completing various visual document understanding
(VDU) tasks, e.g., question answering and information extraction, on real-world
documents through human-written instructions. To this end, we propose
InstructDoc, the first large-scale collection of 30 publicly available VDU
datasets, each with diverse instructions in a unified format, which covers a
wide range of 12 tasks and includes open document types/formats. Furthermore,
to enhance the generalization performance on VDU tasks, we design a new
instruction-based document reading and understanding model, InstructDr, that
connects document images, image encoders, and large language models (LLMs)
through a trainable bridging module. Experiments demonstrate that InstructDr
can effectively adapt to new VDU datasets, tasks, and domains via given
instructions and outperforms existing multimodal LLMs and ChatGPT without
specific training.

我们研究通过人工编写的指令，在真实世界的文档上完成各种视觉文档理解（VDU）任务，例如问答和信息提取。为此，我们提出了 InstructDoc，这是第一个包含 30 个公开可用的 VDU 数据集的大规模集合，每个数据集都有统一格式的多样指令，涵盖了 12 项不同任务并包括多种文档类型 / 格式。此外，为了提高 VDU 任务的泛化性能，我们设计了一种新的基于指令的文档阅读和理解模型 InstructDr，通过可训练的桥接模块连接了文档图像，图像编码器和大型语言模型（LLMs）。实验证明，通过给定的指令，InstructDr 能够有效适应新的 VDU 数据集、任务和领域，且胜过现有的多模态 LLMs 和 ChatGPT 无需特定训练。