Document Information Extraction (DIE) has attracted increasing attention due
to its various advanced applications in the real world. Although recent
literature has already achieved competitive results, these approaches usually
fail when dealing with complex documents with noisy OCR results or mutative
layouts. This paper proposes Generative Multi-modal Network (GMN) for
real-world scenarios to address these problems, which is a robust multi-modal
generation method without predefined label categories. With the carefully
designed spatial encoder and modal-aware mask module, GMN can deal with complex
documents that are hard to serialized into sequential order. Moreover, GMN
tolerates errors in OCR results and requires no character-level annotation,
which is vital because fine-grained annotation of numerous documents is
laborious and even requires annotators with specialized domain knowledge.
Extensive experiments show that GMN achieves new state-of-the-art performance
on several public DIE datasets and surpasses other methods by a large margin,
especially in realistic scenes.

该论文提出了一种面向实际场景的多模态生成方法 GMN，利用空间编码器和模态感知掩模模块处理复杂文档的噪声 OCR 结果或可变布局，并避免了字符级注释，实验证明 GMN 在公共 DIE 数据集上取得了新的最佳性能，并且在现实场景中优于其他方法。