We revisit the "dataset classification" experiment suggested by Torralba and
Efros a decade ago, in the new era with large-scale, diverse, and hopefully
less biased datasets as well as more capable neural network architectures.
Surprisingly, we observe that modern neural networks can achieve excellent
accuracy in classifying which dataset an image is from: e.g., we report 84.7%
accuracy on held-out validation data for the three-way classification problem
consisting of the YFCC, CC, and DataComp datasets. Our further experiments show
that such a dataset classifier could learn semantic features that are
generalizable and transferable, which cannot be simply explained by
memorization. We hope our discovery will inspire the community to rethink the
issue involving dataset bias and model capabilities.

我们重新审视了 Torralba 和 Efros 在十年前建议的 “数据集分类” 实验，这个实验在如今大规模、多样化且希望具有更高的数据集的新时代以及更强大的神经网络架构中进行。令人惊讶的是，我们观察到现代神经网络在分类图像来自哪个数据集的问题上能够达到极高的准确率：例如，在 YFCC、CC 和 DataComp 数据集的三类分类问题的验证数据上，我们报告了 84.7% 的准确率。我们的进一步实验表明，这样的数据集分类器可以学习到具有泛化性和可转移性的语义特征，而这不能简单通过记忆来解释。我们希望我们的发现能够激发社区重新思考涉及数据集偏差和模型能力的问题。