Benchmark datasets play a central role in the organization of machine
learning research. They coordinate researchers around shared research problems
and serve as a measure of progress towards shared goals. Despite the
foundational role of benchmarking practices in this field, relatively little
attention has been paid to the dynamics of benchmark dataset use and reuse,
within or across machine learning subcommunities. In this paper, we dig into
these dynamics. We study how dataset usage patterns differ across machine
learning subcommunities and across time from 2015-2020. We find increasing
concentration on fewer and fewer datasets within task communities, significant
adoption of datasets from other tasks, and concentration across the field on
datasets that have been introduced by researchers situated within a small
number of elite institutions. Our results have implications for scientific
evaluation, AI ethics, and equity/access within the field.

本文研究各机器学习子社区在 2015-2020 年期间数据集使用模式的差异，并发现技术社区对越来越少的数据集集中使用，大量采用来自其他任务的数据集，并集中于由少数精英机构研究者推出的数据集，从而对科学评估、人工智能伦理和公平性、以及领域内的平等 / 可接近性产生影响。