Advances in the Visually-rich Document Understanding (VrDU) field and
particularly the Key-Information Extraction (KIE) task are marked with the
emergence of efficient Transformer-based approaches such as the LayoutLM
models. Despite the good performance of KIE models when fine-tuned on public
benchmarks, they still struggle to generalize on complex real-life use-cases
lacking sufficient document annotations. Our research highlighted that KIE
standard benchmarks such as SROIE and FUNSD contain significant similarity
between training and testing documents and can be adjusted to better evaluate
the generalization of models. In this work, we designed experiments to quantify
the information redundancy in public benchmarks, revealing a 75% template
replication in SROIE official test set and 16% in FUNSD. We also proposed
resampling strategies to provide benchmarks more representative of the
generalization ability of models. We showed that models not suited for document
analysis struggle on the adjusted splits dropping on average 10,5% F1 score on
SROIE and 3.5% on FUNSD compared to multi-modal models dropping only 7,5% F1 on
SROIE and 0.5% F1 on FUNSD.

本研究通过对公共基准数据集 SROIE 和 FUNSD 进行实验，揭示了这两个数据集中存在的训练和测试文件的显著相似性，并提出了重采样策略以更好地评价模型对于具有不同文档注释的真实用例的泛化能力。模型在调整后的划分上效果下降 10.5% 至 SROIE 和 3.5% 至 FUNSD。