Understanding visually-rich business documents to extract structured data and
automate business workflows has been receiving attention both in academia and
industry. Although recent multi-modal language models have achieved impressive
results, we find that existing benchmarks do not reflect the complexity of real
documents seen in industry. In this work, we identify the desiderata for a more
comprehensive benchmark and propose one we call Visually Rich Document
Understanding (VRDU). VRDU contains two datasets that represent several
challenges: rich schema including diverse data types as well as nested
entities, complex templates including tables and multi-column layouts, and
diversity of different layouts (templates) within a single document type. We
design few-shot and conventional experiment settings along with a carefully
designed matching algorithm to evaluate extraction results. We report the
performance of strong baselines and three observations: (1) generalizing to new
document templates is very challenging, (2) few-shot performance has a lot of
headroom, and (3) models struggle with nested fields such as line-items in an
invoice. We plan to open source the benchmark and the evaluation toolkit. We
hope this helps the community make progress on these challenging tasks in
extracting structured data from visually rich documents.

本文提出一个名为 VRDU 的用于评价文档中网络结构、多样的数据类型以及丰富的模板和布局等不同因素综合作用的数据集，并在此基础上设计实验模型，评估文档中结构化数据提取的性能。作者发现，此类任务中面对新模板挑战较大，而针对该问题，本文也提出了一种 few-shot learning 的方法。