Deployed machine learning (ML) models often encounter new user data that
differs from their training data. Therefore, estimating how well a given model
might perform on the new data is an important step toward reliable ML
applications. This is very challenging, however, as the data distribution can
change in flexible ways, and we may not have any labels on the new data, which
is often the case in monitoring settings. In this paper, we propose a new
distribution shift model, Sparse Joint Shift (SJS), which considers the joint
shift of both labels and a few features. This unifies and generalizes several
existing shift models including label shift and sparse covariate shift, where
only marginal feature or label distribution shifts are considered. We describe
mathematical conditions under which SJS is identifiable. We further propose
SEES, an algorithmic framework to characterize the distribution shift under SJS
and to estimate a model's performance on new data without any labels. We
conduct extensive experiments on several real-world datasets with various ML
models. Across different datasets and distribution shifts, SEES achieves
significant (up to an order of magnitude) shift estimation error improvements
over existing approaches.

该文研究了机器学习模型在遇到新的用户数据时，如何估计模型的性能，提出了一种新的分布偏移模型 SJS 和算法框架 SEES，实验结果表明 SEES 在各种数据集和分布偏移情况下，相比现有方法，均能显著提高分布偏移误差的估计精度。