This paper highlights the need to bring document classification benchmarking
closer to real-world applications, both in the nature of data tested ($X$:
multi-channel, multi-paged, multi-industry; $Y$: class distributions and label
set variety) and in classification tasks considered ($f$: multi-page document,
page stream, and document bundle classification, ...). We identify the lack of
public multi-page document classification datasets, formalize different
classification tasks arising in application scenarios, and motivate the value
of targeting efficient multi-page document representations. An experimental
study on proposed multi-page document classification datasets demonstrates that
current benchmarks have become irrelevant and need to be updated to evaluate
complete documents, as they naturally occur in practice. This reality check
also calls for more mature evaluation methodologies, covering calibration
evaluation, inference complexity (time-memory), and a range of realistic
distribution shifts (e.g., born-digital vs. scanning noise, shifting page
order). Our study ends on a hopeful note by recommending concrete avenues for
future improvements.}

本文强调将文档分类基准测试更贴近实际应用的需求，包括在测试的数据性质 ($X$：多通道、多页、多行业；$Y$：类别分布和标签集的多样性) 以及所考虑的分类任务 ($f$：多页文档、页面流和文档捆绑分类)；我们确定了公共多页文档分类数据集的缺乏，形式化了应用场景中产生的不同分类任务，并提出了针对高效多页文档表示的价值的动机；对所提出的多页文档分类数据集进行的实证研究表明，当前的基准测试已经不再相关，需要更新以评估实际中出现的完整文档；这种现实检验也呼吁采用更成熟的评估方法，包括校准评估、推理复杂性（时间 - 内存）和一系列现实分布变化（如 born-digital vs. 扫描噪音、页面次序变化）；我们研究以对未来改进提出具体建议的希望之声作为结束。