We present an end-to-end, multimodal, fully convolutional network for extracting semantic structures from document images. We consider document semantic structure extraction as a pixel-wise segmentation task, and propose a unified model that classifies pixels based not only on their visual appearance, as in the traditional page segmentation task, but also on the content of underlying text. Moreover, we propose an efficient synthetic document generation process that we use to generate pretraining data for our network. Once the network is trained on a large set of synthetic documents, we fine-tune the network on unlabeled real documents using a semi-supervised approach. We systematically study the optimum network architecture and show that both our multimodal approach and the synthetic data pretraining significantly boost the performance.

该研究提出了一种全卷积网络，通过多模态方法从文档图像中提取语义结构，并使用像素级分割方式考虑文档的语义结构提取。该方法不仅考虑文本的视觉外观，还考虑文本的内容，通过有效的合成文档生成数据进行预训练和半监督学习方法滴定优化网络架构，施加于真实文档上，此方法极大地提高了分割的性能表现。

使用多模态全卷积神经网络学习从文档中提取语义结构