Understanding documents is central to many real-world tasks but remains a
challenging topic. Unfortunately, there is no well-established consensus on how
to comprehensively evaluate document understanding abilities, which
significantly hinders the fair comparison and measuring the progress of the
field. To benchmark document understanding researches, this paper summarizes
four representative abilities, i.e., document classification, document
structural analysis, document information extraction, and document
transcription. Under the new evaluation framework, we propose \textbf{Document
Language Understanding Evaluation} -- \textbf{DLUE}, a new task suite which
covers a wide-range of tasks in various forms, domains and document genres. We
also systematically evaluate six well-established transformer models on DLUE,
and find that due to the lengthy content, complicated underlying structure and
dispersed knowledge, document understanding is still far from being solved, and
currently there is no neural architecture that dominates all tasks, raising
requirements for a universal document understanding architecture.

本文提出一种新的文档语言理解评测框架 ——DLUE，它涵盖了各种形式、领域和文档类型的文档理解任务，并对六个知名 Transformer 模型进行了系统评估，结果显示由于内容冗长、结构复杂和知识分散，文档理解仍然远未解决，没有一种神经架构能够支配所有任务，这需要一个通用的文档理解结构。