Recently, leveraging large language models (LLMs) or multimodal large
language models (MLLMs) for document understanding has been proven very
promising. However, previous works that employ LLMs/MLLMs for document
understanding have not fully explored and utilized the document layout
information, which is vital for precise document understanding. In this paper,
we propose LayoutLLM, an LLM/MLLM based method for document understanding. The
core of LayoutLLM is a layout instruction tuning strategy, which is specially
designed to enhance the comprehension and utilization of document layouts. The
proposed layout instruction tuning strategy consists of two components:
Layout-aware Pre-training and Layout-aware Supervised Fine-tuning. To capture
the characteristics of document layout in Layout-aware Pre-training, three
groups of pre-training tasks, corresponding to document-level, region-level and
segment-level information, are introduced. Furthermore, a novel module called
layout chain-of-thought (LayoutCoT) is devised to enable LayoutLLM to focus on
regions relevant to the question and generate accurate answers. LayoutCoT is
effective for boosting the performance of document understanding. Meanwhile, it
brings a certain degree of interpretability, which could facilitate manual
inspection and correction. Experiments on standard benchmarks show that the
proposed LayoutLLM significantly outperforms existing methods that adopt
open-source 7B LLMs/MLLMs for document understanding. The training data of the
LayoutLLM is publicly available at
this https URL

本研究提出了一种基于 LLM / MLLM 的文档理解方法 LayoutLLM，核心是一种专门设计的布局指令调整策略，通过布局感知的预训练和布局感知的监督微调来提高对文档布局的理解和利用，同时使用 LayoutCoT 模块来生成准确答案并提升文档理解的性能。