The growing prevalence of visually rich documents, such as webpages and
scanned/digital-born documents (images, PDFs, etc.), has led to increased
interest in automatic document understanding and information extraction across
academia and industry. Although various document modalities, including image,
text, layout, and structure, facilitate human information retrieval, the
interconnected nature of these modalities presents challenges for neural
networks. In this paper, we introduce WebLM, a multimodal pre-training network
designed to address the limitations of solely modeling text and structure
modalities of HTML in webpages. Instead of processing document images as
unified natural images, WebLM integrates the hierarchical structure of document
images to enhance the understanding of markup-language-based documents.
Additionally, we propose several pre-training tasks to model the interaction
among text, structure, and image modalities effectively. Empirical results
demonstrate that the pre-trained WebLM significantly surpasses previous
state-of-the-art pre-trained models across several webpage understanding tasks.
The pre-trained models and code are available at
this https URL

通过整合文本、结构和图像等多模态信息，使用多模态预训练网络进行自动文档理解和信息提取，实现了对网页的深入理解，显著提高了网页理解任务的性能。