We might hope that when faced with unexpected inputs, well-designed software
systems would fire off warnings. Machine learning (ML) systems, however, which
depend strongly on properties of their inputs (e.g. the i.i.d. assumption),
tend to fail silently. This paper explores the problem of building ML systems
that fail loudly, investigating methods for detecting dataset shift,
identifying exemplars that most typify the shift, and quantifying shift
malignancy. We focus on several datasets and various perturbations to both
covariates and label distributions with varying magnitudes and fractions of
data affected. Interestingly, we show that across the dataset shifts that we
explore, a two-sample-testing-based approach, using pre-trained classifiers for
dimensionality reduction, performs best. Moreover, we demonstrate that
domain-discriminating approaches tend to be helpful for characterizing shifts
qualitatively and determining if they are harmful.

本文探讨机器学习系统在面对各类输入时，如何通过检测数据集漂移并量化其恶性，从而建立高鲁棒性的系统，同时提出使用预训练分类器的双样本测试法在多种数据集漂移情况下表现最佳，且判别方法有助于定性确定数据集漂移的影响。