Benchmark datasets play a central role in the organization of machine
learning research. They coordinate researchers around shared research problems
and serve as a measure of progress towards shared goals. Despite the
foundational role of benchmarking practices in this field, relatively little
attention has been paid to the dynamics of benchmark dataset use and reuse,
within or across machine learning subcommunities. In this paper, we dig into
these dynamics. We study how dataset usage patterns differ across machine
learning subcommunities and across time from 2015-2020. We find increasing
concentration on fewer and fewer datasets within task communities, significant
adoption of datasets from other tasks, and concentration across the field on
datasets that have been introduced by researchers situated within a small
number of elite institutions. Our results have implications for scientific
evaluation, AI ethics, and equity/access within the field.

本文研究各机器学习子社区在 2015-2020 年期间数据集使用模式的差异，并发现技术社区对越来越少的数据集集中使用，大量采用来自其他任务的数据集，并集中于由少数精英机构研究者推出的数据集，从而对科学评估、人工智能伦理和公平性、以及领域内的平等 / 可接近性产生影响。

机器学习研究中数据集的生命周期：减少、重复利用和循环利用

Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning  Research

The most significant barrier to the advancement of Neural Architecture Search
(NAS) is its demand for large computational resources, which hinders
scientifically sound empirical evaluations of NAS methods. Tabular NAS
benchmarks have alleviated this problem substantially, making it possible to
properly evaluate NAS methods in seconds on commodity machines. However, an
unintended consequence of tabular NAS benchmarks has been a focus on extremely
small architectural search spaces since their construction relies on exhaustive
evaluations of the space. This leads to unrealistic results that do not
transfer to larger spaces. To overcome this fundamental limitation, we propose
a methodology to create cheap NAS surrogate benchmarks for arbitrary search
spaces. We exemplify this approach by creating surrogate NAS benchmarks on the
existing tabular NAS-Bench-101 and on two widely used NAS search spaces with up
to $10^{21}$ architectures ($10^{13}$ times larger than any previous tabular
NAS benchmark). We show that surrogate NAS benchmarks can model the true
performance of architectures better than tabular benchmarks (at a small
fraction of the cost), that they lead to faithful estimates of how well
different NAS methods work on the original non-surrogate benchmark, and that
they can generate new scientific insight. We open-source all our code and
believe that surrogate NAS benchmarks are an indispensable tool to extend
scientifically sound work on NAS to large and exciting search spaces.

为了解决神经结构搜索方法需要大量计算资源的问题，提出了一种使用代理 NAS 基准来对任意搜索空间进行成本相对较低的科学评估的方法，并证明代理 NAS 基准可以更好地建模架构的真实性能，以及扩展了 NAS 领域的科学研究范围。

代理 NAS 基准测试：超越表格 NAS 基准测试的有限搜索空间

Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of  Tabular NAS Benchmarks

Modern bibliographic databases provide the basis for scientific research and
its evaluation. While their content and structure differ substantially, there
exist only informal notions on their reliability. Here we compare the
topological consistency of citation networks extracted from six popular
bibliographic databases including Web of Science, CiteSeer and arXiv.org. The
networks are assessed through a rich set of local and global graph statistics.
We first reveal statistically significant inconsistencies between some of the
databases with respect to individual statistics. For example, the introduced
field bow-tie decomposition of DBLP Computer Science Bibliography substantially
differs from the rest due to the coverage of the database, while the citation
information within arXiv.org is the most exhaustive. Finally, we compare the
databases over multiple graph statistics using the critical difference diagram.
The citation topology of DBLP Computer Science Bibliography is the least
consistent with the rest, while, not surprisingly, Web of Science is
significantly more reliable from the perspective of consistency. This work can
serve either as a reference for scholars in bibliometrics and scientometrics or
a scientific evaluation guideline for governments and research agencies.

本文比较了 6 个常用的文献数据库，通过图统计学的方法比较它们的引文网络的拓扑一致性，并发现其中存在一些统计不一致的现象。对比结果发现， DBLP 计算机科学文献库的引文网络的一致性最差，而 Web of Science 从一致性的角度来看更可靠。这项工作既可以作为文献计量学和科学计量学领域学者的参考，也可以作为政府和研究机构科学评估的指导方针。