For machine learning systems to be reliable, we must understand their
performance in unseen, out-of-distribution environments. In this paper, we
empirically show that out-of-distribution performance is strongly correlated
with in-distribution performance for a wide range of models and distribution
shifts. Specifically, we demonstrate strong correlations between
in-distribution and out-of-distribution performance on variants of CIFAR-10 &
ImageNet, a synthetic pose estimation task derived from YCB objects, satellite
imagery classification in FMoW-WILDS, and wildlife classification in
iWildCam-WILDS. The strong correlations hold across model architectures,
hyperparameters, training set size, and training duration, and are more precise
than what is expected from existing domain adaptation theory. To complete the
picture, we also investigate cases where the correlation is weaker, for
instance some synthetic distribution shifts from CIFAR-10-C and the tissue
classification dataset Camelyon17-WILDS. Finally, we provide a candidate theory
based on a Gaussian data model that shows how changes in the data covariance
arising from distribution shift can affect the observed correlations.

通过对 CIFAR-10、ImageNet、FMoW-WILDS 和 iWildCam-WILDS 等模型和数据集的实证研究，我们展示了机器学习系统在未知、分布不同的环境中表现的强相关性，并提供了基于高斯数据模型的理论解释。