Structure information is critical for understanding the semantics of
text-rich images, such as documents, tables, and charts. Existing Multimodal
Large Language Models (MLLMs) for Visual Document Understanding are equipped
with text recognition ability but lack general structure understanding
abilities for text-rich document images. In this work, we emphasize the
importance of structure information in Visual Document Understanding and
propose the Unified Structure Learning to boost the performance of MLLMs. Our
Unified Structure Learning comprises structure-aware parsing tasks and
multi-grained text localization tasks across 5 domains: document, webpage,
table, chart, and natural image. To better encode structure information, we
design a simple and effective vision-to-text module H-Reducer, which can not
only maintain the layout information but also reduce the length of visual
features by merging horizontal adjacent patches through convolution, enabling
the LLM to understand high-resolution images more efficiently. Furthermore, by
constructing structure-aware text sequences and multi-grained pairs of texts
and bounding boxes for publicly available text-rich images, we build a
comprehensive training set DocStruct4M to support structure learning. Finally,
we construct a small but high-quality reasoning tuning dataset DocReason25K to
trigger the detailed explanation ability in the document domain. Our model
DocOwl 1.5 achieves state-of-the-art performance on 10 visual document
understanding benchmarks, improving the SOTA performance of MLLMs with a 7B LLM
by more than 10 points in 5/10 benchmarks. Our codes, models, and datasets are
publicly available at
this https URL

利用结构信息提升多模态大型语言模型在视觉文档理解中的性能，通过构建一种新的统一结构学习方法，设计了一个简单而有效的 H-Reducer 视觉到文本模块，并构建了相应的训练数据集和细致解释能力的推理调优数据集，提升了视觉文档理解基准任务的性能。