Machine learning systems deployed in the wild are often trained on a source
distribution but deployed on a different target distribution. Unlabeled data
can be a powerful point of leverage for mitigating these distribution shifts,
as it is frequently much more available than labeled data and can often be
obtained from distributions beyond the source distribution as well. However,
existing distribution shift benchmarks with unlabeled data do not reflect the
breadth of scenarios that arise in real-world applications. In this work, we
present the WILDS 2.0 update, which extends 8 of the 10 datasets in the WILDS
benchmark of distribution shifts to include curated unlabeled data that would
be realistically obtainable in deployment. These datasets span a wide range of
applications (from histology to wildlife conservation), tasks (classification,
regression, and detection), and modalities (photos, satellite images,
microscope slides, text, molecular graphs). The update maintains consistency
with the original WILDS benchmark by using identical labeled training,
validation, and test sets, as well as the evaluation metrics. On these
datasets, we systematically benchmark state-of-the-art methods that leverage
unlabeled data, including domain-invariant, self-training, and self-supervised
methods, and show that their success on WILDS is limited. To facilitate method
development and evaluation, we provide an open-source package that automates
data loading and contains all of the model architectures and methods used in
this paper. Code and leaderboards are available at this https URL

本篇论文介绍了 WILDS 2.0 的更新，通过增加真实部署中可获得的无标签数据，扩展了 WILDS 基准测试中 10 个数据集中的 8 个。研究人员在这些数据集上进行了系统性基准测试，展示了当前最先进的方法使用无标签数据在现实世界应用中的局限性，并提供了一个开源软件包来自动化数据加载和包含本文中使用的所有模型架构和方法。