Document intelligence automates the extraction of information from documents
and supports many business applications. Recent self-supervised learning
methods on large-scale unlabeled document datasets have opened up promising
directions towards reducing annotation efforts by training models with
self-supervised objectives. However, most of the existing document pretraining
methods are still language-dominated. We present UDoc, a new unified
pretraining framework for document understanding. UDoc is designed to support
most document understanding tasks, extending the Transformer to take multimodal
embeddings as input. Each input element is composed of words and visual
features from a semantic region of the input document image. An important
feature of UDoc is that it learns a generic representation by making use of
three self-supervised losses, encouraging the representation to model
sentences, learn similarities, and align modalities. Extensive empirical
analysis demonstrates that the pretraining procedure learns better joint
representations and leads to improvements in downstream tasks.

UDoc 是一个新的文档理解的统一预训练框架，它通过使用三个自监督损失约束表示模型，将单词和视觉特征作为输入，使神经网络从无标签的数据中学习到更好的文档表示，并取得了下游任务的提升。