Building rich machine learning datasets in a scalable manner often
necessitates a crowd-sourced data collection pipeline. In this work, we use
human studies to investigate the consequences of employing such a pipeline,
focusing on the popular ImageNet dataset. We study how specific design choices
in the ImageNet creation process impact the fidelity of the resulting
dataset---including the introduction of biases that state-of-the-art models
exploit. Our analysis pinpoints how a noisy data collection pipeline can lead
to a systematic misalignment between the resulting benchmark and the real-world
task it serves as a proxy for. Finally, our findings emphasize the need to
augment our current model training and evaluation toolkit to take such
misalignments into account. To facilitate further research, we release our
refined ImageNet annotations at this https URL

通过人工研究这项工作研究了采用众包数据收集管道构建丰富的机器学习数据集的后果，重点关注了流行的 ImageNet 数据集，在 ImageNet 创建过程中特定的设计选择如何影响所得到的数据集的保真度，以及噪声数据收集管道如何导致所得基准与其代理真实世界任务之间存在系统性偏差。最后，发现需要扩充我们的模型培训和评估工具包，以考虑这种不对齐现象。