Visually-rich document entity retrieval (VDER), which extracts key
information (e.g. date, address) from document images like invoices and
receipts, has become an important topic in industrial NLP applications. The
emergence of new document types at a constant pace, each with its unique entity
types, presents a unique challenge: many documents contain unseen entity types
that occur only a couple of times. Addressing this challenge requires models to
have the ability of learning entities in a few-shot manner. However, prior
works for Few-shot VDER mainly address the problem at the document level with a
predefined global entity space, which doesn't account for the entity-level
few-shot scenario: target entity types are locally personalized by each task
and entity occurrences vary significantly among documents. To address this
unexplored scenario, this paper studies a novel entity-level few-shot VDER
task. The challenges lie in the uniqueness of the label space for each task and
the increased complexity of out-of-distribution (OOD) contents. To tackle this
novel task, we present a task-aware meta-learning based framework, with a
central focus on achieving effective task personalization that distinguishes
between in-task and out-of-task distribution. Specifically, we adopt a
hierarchical decoder (HC) and employ contrastive learning (ContrastProtoNet) to
achieve this goal. Furthermore, we introduce a new dataset, FewVEX, to boost
future research in the field of entity-level few-shot VDER. Experimental
results demonstrate our approaches significantly improve the robustness of
popular meta-learning baselines.

本研究旨在解决实体级少样本视觉文档实体检索 (VDER) 任务的挑战，并通过引入任务感知的元学习框架以及新的数据集 FewVEX，显著提高了流行的元学习基准模型的鲁棒性。

面向任务个性化的多模态少样本学习在视觉丰富的文档实体检索中的应用

On Task-personalized Multimodal Few-shot Learning for Visually-rich  Document Entity Retrieval

Visually-Rich Document Entity Retrieval (VDER) is a type of machine learning
task that aims at recovering text spans in the documents for each of the
entities in question. VDER has gained significant attention in recent years
thanks to its broad applications in enterprise AI. Unfortunately, as document
images often contain personally identifiable information (PII), publicly
available data have been scarce, not only because of privacy constraints but
also the costs of acquiring annotations. To make things worse, each dataset
would often define its own sets of entities, and the non-overlapping entity
spaces between datasets make it difficult to transfer knowledge between
documents. In this paper, we propose a method to collect massive-scale, noisy,
and weakly labeled data from the web to benefit the training of VDER models.
Such a method will generate a huge amount of document image data to compensate
for the lack of training data in many VDER settings. Moreover, the collected
dataset named DocuNet would not need to be dependent on specific document types
or entity sets, making it universally applicable to all VDER tasks. Empowered
by DocuNet, we present a lightweight multimodal architecture named UniFormer,
which can learn a unified representation from text, layout, and image crops
without needing extra visual pertaining. We experiment with our methods on
popular VDER models in various settings and show the improvements when this
massive dataset is incorporated with UniFormer on both classic entity retrieval
and few-shot learning settings.

本文提出了一种从网络上收集海量、嘈杂、弱标注数据的方法，以弥补许多 VDER 设置中训练数据不足的缺陷，也给出了一个名为 DocuNet 的收集数据集，它不需要依赖特定文档类型或实体集，从而在各种 VDER 任务中通用应用。借助 DocuNet，我们提出了一种轻量级多模态体系结构 UniFormer，在没有额外的视觉相关性的情况下从文本、布局和图像裁剪中学习统一的表征。在不同的设置中实验我们的方法并展示了在传统实体检索和少样本学习设置中，当将这个大规模数据集与 UniFormer 相结合时的改进。