Recently developed pre-trained text-and-layout models (PTLMs) have shown
remarkable success in multiple information extraction tasks on visually-rich
documents. However, the prevailing evaluation pipeline may not be sufficiently
robust for assessing the information extraction ability of PTLMs, due to
inadequate annotations within the benchmarks. Therefore, we claim the necessary
standards for an ideal benchmark to evaluate the information extraction ability
of PTLMs. We then introduce EC-FUNSD, an entity-centric benckmark designed for
the evaluation of semantic entity recognition and entity linking on
visually-rich documents. This dataset contains diverse formats of document
layouts and annotations of semantic-driven entities and their relations.
Moreover, this dataset disentangles the falsely coupled annotation of segment
and entity that arises from the block-level annotation of FUNSD. Experiment
results demonstrate that state-of-the-art PTLMs exhibit overfitting tendencies
on the prevailing benchmarks, as their performance sharply decrease when the
dataset bias is removed.

我们介绍了一个用于评估 PTLMs 的信息提取能力的理想基准，并引入了 EC-FUNSD，一个用于评估语义实体识别和实体链接的实体中心基准，该数据集包含多种文档布局格式和语义驱动实体及其关系的注释。实验结果表明，最先进的 PTLMs 在现有基准上存在过拟合趋势，去除数据集偏差后，它们的性能明显下降。