Deployment of machine learning (ML) algorithms in production for extended
periods of time has uncovered new challenges such as monitoring and management
of real-time prediction quality of a model in the absence of labels. However,
such tracking is imperative to prevent catastrophic business outcomes resulting
from incorrect predictions. The scale of these deployments makes manual
monitoring prohibitive, making automated techniques to track and raise alerts
imperative. We present a framework, ML Health, for tracking potential drops in
the predictive performance of ML models in the absence of labels. The framework
employs diagnostic methods to generate alerts for further investigation. We
develop one such method to monitor potential problems when production data
patterns do not match training data distributions. We demonstrate that our
method performs better than standard "distance metrics", such as RMSE,
KL-Divergence, and Wasserstein at detecting issues with mismatched data sets.
Finally, we present a working system that incorporates the ML Health approach
to monitor and manage ML deployments within a realistic full production ML
lifecycle.

ML Health 为监测机器学习模型预测性能潜在下降提出了一种诊断方法，并将该方法应用于一种实现了全生产生命周期的系统中，来自动化追踪并生成进一步调查的警报。