Machine learning models are essential tools in various domains, but their
performance can degrade over time due to changes in data distribution or other
factors. On one hand, detecting and addressing such degradations is crucial for
maintaining the models' reliability. On the other hand, given enough data, any
arbitrary small change of quality can be detected. As interventions, such as
model re-training or replacement, can be expensive, we argue that they should
only be carried out when changes exceed a given threshold. We propose a
sequential monitoring scheme to detect these relevant changes. The proposed
method reduces unnecessary alerts and overcomes the multiple testing problem by
accounting for temporal dependence of the measured model quality. Conditions
for consistency and specified asymptotic levels are provided. Empirical
validation using simulated and real data demonstrates the superiority of our
approach in detecting relevant changes in model quality compared to benchmark
methods. Our research contributes a practical solution for distinguishing
between minor fluctuations and meaningful degradations in machine learning
model performance, ensuring their reliability in dynamic environments.

该研究提出了一种序列监测方案，通过考虑测量模型质量的时间依赖性，减少不必要的警报并解决多重测试问题，从而在检测模型质量相关变化方面优于基准方法。此研究为在动态环境中区分小幅波动和有意义的模型性能退化提供了实用解决方案，确保机器学习模型的可靠性。

监控机器学习模型：在线检测相关偏差

Monitoring Machine Learning Models: Online Detection of Relevant  Deviations

In this work we give a case study of an embodied machine-learning (ML)
powered agent that improves itself via interactions with crowd-workers. The
agent consists of a set of modules, some of which are learned, and others
heuristic. While the agent is not "end-to-end" in the ML sense, end-to-end
interaction is a vital part of the agent's learning mechanism. We describe how
the design of the agent works together with the design of multiple annotation
interfaces to allow crowd-workers to assign credit to module errors from
end-to-end interactions, and to label data for individual modules. Over
multiple automated human-agent interaction, credit assignment, data annotation,
and model re-training and re-deployment, rounds we demonstrate agent
improvement.

本文介绍了一个基于身体型机器学习的代理系统，通过与人群工作者互动，改进自身。我们通过多次自动化的人 - 代理交互，学分分配，数据注释，重训练和重新部署模型来证明代理改进。