We propose SelfDoc, a task-agnostic pre-training framework for document image understanding. Because documents are multimodal and are intended for sequential reading, our framework exploits the positional, textual, and visual information of every semantically meaningful component in a document, and it models the contextualization between each block of content. Unlike existing document pre-training models, our model is coarse-grained instead of treating individual words as input, therefore avoiding an overly fine-grained with excessive contextualization. Beyond that, we introduce cross-modal learning in the model pre-training phase to fully leverage multimodal information from unlabeled documents. For downstream usage, we propose a novel modality-adaptive attention mechanism for multimodal feature fusion by adaptively emphasizing language and vision signals. Our framework benefits from self-supervised pre-training on documents without requiring annotations by a feature masking training strategy. It achieves superior performance on multiple downstream tasks with significantly fewer document images used in the pre-training stage compared to previous works.

SelfDoc是一个文档图像理解的任务无关的预训练框架，利用文档的位置、文本和视觉信息，并建模内容块之间的上下文关系，提出了一种新的跨模态学习模型，优于现有模型，同时具有自适应的视觉语言融合机制并应用自监督模型预训练，与以前作品相比，使用较少的文档图片达到更好的性能。

SelfDoc: 自我监督文件表示学习