Amid rising concerns of reproducibility and generalizability in predictive
modeling, we explore the possibility and potential benefits of introducing
pre-registration to the field. Despite notable advancements in predictive
modeling, spanning core machine learning tasks to various scientific
applications, challenges such as overlooked contextual factors, data-dependent
decision-making, and unintentional re-use of test data have raised questions
about the integrity of results. To address these issues, we propose adapting
pre-registration practices from explanatory modeling to predictive modeling. We
discuss current best practices in predictive modeling and their limitations,
introduce a lightweight pre-registration template, and present a qualitative
study with machine learning researchers to gain insight into the effectiveness
of pre-registration in preventing biased estimates and promoting more reliable
research outcomes. We conclude by exploring the scope of problems that
pre-registration can address in predictive modeling and acknowledging its
limitations within this context.

通过研究预测建模中出现的再现性和泛化性的担忧，我们探讨引入先注册的可能性和潜在好处。虽然预测建模取得了显著进展，涵盖了核心机器学习任务和各种科学应用，但是忽视的背景因素、基于数据的决策和意外重复使用测试数据等挑战引发了对结果完整性的质疑。为解决这些问题，我们建议从解释性建模中借鉴先注册实践到预测建模中。我们讨论了预测建模的当前最佳实践及其局限性，引入了一个轻量级的先注册模板，并进行了一项定性研究，以了解先注册在预防有偏估计和促进更可靠的研究结果方面的有效性。最后，我们探讨了先注册可以解决的预测建模问题的范围，并在此背景下承认其局限性。

预测建模的预登记

Pre-registration for Predictive Modeling

Estimating the generalization error (GE) of machine learning models is
fundamental, with resampling methods being the most common approach. However,
in non-standard settings, particularly those where observations are not
independently and identically distributed, resampling using simple random data
divisions may lead to biased GE estimates. This paper strives to present
well-grounded guidelines for GE estimation in various such non-standard
settings: clustered data, spatial data, unequal sampling probabilities, concept
drift, and hierarchically structured outcomes. Our overview combines
well-established methodologies with other existing methods that, to our
knowledge, have not been frequently considered in these particular settings. A
unifying principle among these techniques is that the test data used in each
iteration of the resampling procedure should reflect the new observations to
which the model will be applied, while the training data should be
representative of the entire data set used to obtain the final model. Beyond
providing an overview, we address literature gaps by conducting simulation
studies. These studies assess the necessity of using GE-estimation methods
tailored to the respective setting. Our findings corroborate the concern that
standard resampling methods often yield biased GE estimates in non-standard
settings, underscoring the importance of tailored GE estimation.

在非标准设置中，本文试图提供针对一般化误差在集群数据、空间数据、不均匀采样概率、概念漂移和分层结构结果等各种非标准情况下的估计的深入指南，结合现有方法的综述和模拟研究，揭示了标准重样本方法通常在非标准设置中导致偏倚的问题，强调了定制化的一般化误差估计的重要性。

在非标准环境下评估机器学习模型：概述与新发现

Evaluating machine learning models in non-standard settings: An overview  and new findings

We consider the problem of uncertainty estimation in the context of
(non-Bayesian) deep neural classification. In this context, all known methods
are based on extracting uncertainty signals from a trained network optimized to
solve the classification problem at hand. We demonstrate that such techniques
tend to introduce biased estimates for instances whose predictions are supposed
to be highly confident. We argue that this deficiency is an artifact of the
dynamics of training with SGD-like optimizers, and it has some properties
similar to overfitting. Based on this observation, we develop an uncertainty
estimation algorithm that selectively estimates the uncertainty of highly
confident points, using earlier snapshots of the trained model, before their
estimates are jittered (and way before they are ready for actual
classification). We present extensive experiments indicating that the proposed
algorithm provides uncertainty estimates that are consistently better than all
known methods.

本研究提出了一种基于模型历史快照的算法，用于在非贝叶斯深度神经分类中，有选择地估计高度自信点的不确定性，这解决了从已训练网络中提取不确定信号的已知方法所带来的偏差估计问题，研究表明所提出的算法比所有已知方法的不确定性估计结果更加准确。