Key information extraction (KIE) from scanned documents has gained increasing
attention because of its applications in various domains. Although promising
results have been achieved by some recent KIE approaches, they are usually
built based on discriminative models, which lack the ability to handle optical
character recognition (OCR) errors and require laborious token-level labelling.
In this paper, we propose a novel generative end-to-end model, named GenKIE, to
address the KIE task. GenKIE is a sequence-to-sequence multimodal generative
model that utilizes multimodal encoders to embed visual, layout and textual
features and a decoder to generate the desired output. Well-designed prompts
are leveraged to incorporate the label semantics as the weakly supervised
signals and entice the generation of the key information. One notable advantage
of the generative model is that it enables automatic correction of OCR errors.
Besides, token-level granular annotation is not required. Extensive experiments
on multiple public real-world datasets show that GenKIE effectively generalizes
over different types of documents and achieves state-of-the-art results. Our
experiments also validate the model's robustness against OCR errors, making
GenKIE highly applicable in real-world scenarios.

提出了一种名为 GenKIE 的新型生成式端到端模型，利用多模态编码器嵌入视觉、版面布局和文本特征，并利用解码器生成所需的输出，从而解决了来自扫描文档的关键信息抽取任务。实验证明 GenKIE 能够有效泛化到不同类型的文档，并在 OCR 错误方面表现出鲁棒性，因此在实际场景中具有广泛的应用前景。

GenKIE：鲁棒的生成多模态文档关键信息抽取

GenKIE: Robust Generative Multimodal Document Key Information Extraction

Developing an agent capable of adapting to unseen environments remains a
difficult challenge in imitation learning. In this work, we present Adaptive
Return-conditioned Policy (ARP), an efficient framework designed to enhance the
agent's generalization ability using natural language task descriptions and
pre-trained multimodal encoders. Our key idea is to calculate a similarity
between visual observations and natural language instructions in the
pre-trained multimodal embedding space (such as CLIP) and use it as a reward
signal. We then train a return-conditioned policy using expert demonstrations
labeled with multimodal rewards. Because the multimodal rewards provide
adaptive signals at each timestep, our ARP effectively mitigates the goal
misgeneralization. This results in superior generalization performances even
when faced with unseen text instructions, compared to existing text-conditioned
policies. To improve the quality of rewards, we also introduce a fine-tuning
method for pre-trained multimodal encoders, further enhancing the performance.
Video demonstrations and source code are available on the project website:
this https URL

通过使用自然语言任务描述和预训练的多模态编码器，我们提出了一种名为自适应返回条件策略（ARP）的高效框架，以增强代理的泛化能力，通过在预训练的多模态嵌入空间（如 CLIP）中计算视觉观察和自然语言指令之间的相似度作为奖励信号，并使用它训练具有多模态奖励标签的返回条件策略，从而有效地减轻目标误泛化。即使面对未知的文本指令，相比于现有的文本条件策略，我们的 ARP 通过提供每个时间步骤上的自适应信号，表现出卓越的泛化性能。为了提高奖励的质量，我们还引入了一种用于预训练的多模态编码器的微调方法，进一步增强性能。