This paper presents GenDoc, a general sequence-to-sequence document
understanding model pre-trained with unified masking across three modalities:
text, image, and layout. The proposed model utilizes an encoder-decoder
architecture, which allows for increased adaptability to a wide range of
downstream tasks with diverse output formats, in contrast to the encoder-only
models commonly employed in document understanding. In addition to the
traditional text infilling task used in previous encoder-decoder models, our
pre-training extends to include tasks of masked image token prediction and
masked layout prediction. We also design modality-specific instruction and
adopt both disentangled attention and the mixture-of-modality-experts strategy
to effectively capture the information leveraged by each modality. Evaluation
of the proposed model through extensive experiments on several downstream tasks
in document understanding demonstrates its ability to achieve superior or
competitive performance compared to state-of-the-art approaches. Our analysis
further suggests that GenDoc is more robust than the encoder-only models in
scenarios where the OCR quality is imperfect.

本文提出 GenDoc，这是一个使用序列到序列模型进行预训练的通用文档理解模型，该模型采用编码器 - 解码器架构，可适应多种输出格式的下游任务，通过包括掩码图像令牌预测和掩码布局预测在内的多个任务进行预训练，采用分离的注意力和模态专家策略来有效地捕获每种模态利用的信息，实验结果显示，与最先进的方法相比，该模型具有更强的鲁棒性。