In the realm of machine learning and statistical modeling, practitioners
often work under the assumption of accessible, static, labeled data for
evaluation and training. However, this assumption often deviates from reality
where data may be private, encrypted, difficult- to-measure, or unlabeled. In
this paper, we bridge this gap by adapting the Hui-Walter paradigm, a method
traditionally applied in epidemiology and medicine, to the field of machine
learning. This approach enables us to estimate key performance metrics such as
false positive rate, false negative rate, and priors in scenarios where no
ground truth is available. We further extend this paradigm for handling online
data, opening up new possibilities for dynamic data environments. Our
methodology involves partitioning data into latent classes to simulate multiple
data populations (if natural populations are unavailable) and independently
training models to replicate multiple tests. By cross-tabulating binary
outcomes across ensemble categorizers and multiple populations, we are able to
estimate unknown parameters through Gibbs sampling, eliminating the need for
ground-truth or labeled data. This paper showcases the potential of our
methodology to transform machine learning practices by allowing for accurate
model assessment under dynamic and uncertain data conditions.

该论文介绍了将流行病学和医学领域中传统应用的 Hui-Walter 模式应用于机器学习领域，以估计关键性能指标和处理动态和不确定的数据环境。