The RVL-CDIP benchmark is widely used for measuring performance on the task
of document classification. Despite its widespread use, we reveal several
undesirable characteristics of the RVL-CDIP benchmark. These include (1)
substantial amounts of label noise, which we estimate to be 8.1% (ranging
between 1.6% to 16.9% per document category); (2) presence of many ambiguous or
multi-label documents; (3) a large overlap between test and train splits, which
can inflate model performance metrics; and (4) presence of sensitive
personally-identifiable information like US Social Security numbers (SSNs). We
argue that there is a risk in using RVL-CDIP for benchmarking document
classifiers, as its limited scope, presence of errors (state-of-the-art models
now achieve accuracy error rates that are within our estimated label error
rate), and lack of diversity make it less than ideal for benchmarking. We
further advocate for the creation of a new document classification benchmark,
and provide recommendations for what characteristics such a resource should
include.

揭示了 RVL-CDIP 标准存在标签误差、多标签问题、测试和训练数据重复、存在敏感信息等问题，并建议建立一个新的文件分类基准。