Clinical prediction models are statistical or machine learning models used to
quantify the risk of a certain health outcome using patient data. These can
then inform potential interventions on patients, causing an effect called
performative prediction: predictions inform interventions which influence the
outcome they were trying to predict, leading to a potential underestimation of
risk in some patients if a model is updated on this data. One suggested
resolution to this is the use of hold-out sets, in which a set of patients do
not receive model derived risk scores, such that a model can be safely
retrained. We present an overview of clinical and research ethics regarding
potential implementation of hold-out sets for clinical prediction models in
health settings. We focus on the ethical principles of beneficence,
non-maleficence, autonomy and justice. We also discuss informed consent,
clinical equipoise, and truth-telling. We present illustrative cases of
potential hold-out set implementations and discuss statistical issues arising
from different hold-out set sampling methods. We also discuss differences
between hold-out sets and randomised control trials, in terms of ethics and
statistical issues. Finally, we give practical recommendations for researchers
interested in the use hold-out sets for clinical prediction models.

临床预测模型的伦理研究中探讨了利用保留集进行临床预测模型的潜在实施，涉及到临床预测模型、预测效果、伦理原则、伦理和统计问题。提出了如何选择保留集样本、以及与随机对照试验的伦理和统计差异等问题，并对研究者使用保留集进行临床预测模型提出实用建议。

临床预测模型管理中留置集使用的伦理考虑

Ethical considerations of use of hold-out sets in clinical prediction  model management

Traditionally, machine learning-based clinical prediction models have been
trained and evaluated on patient data from a single source, such as a hospital.
Cross-validation methods can be used to estimate the accuracy of such models on
new patients originating from the same source, by repeated random splitting of
the data. However, such estimates tend to be highly overoptimistic when
compared to accuracy obtained from deploying models to sources not represented
in the dataset, such as a new hospital. The increasing availability of
multi-source medical datasets provides new opportunities for obtaining more
comprehensive and realistic evaluations of expected accuracy through
source-level cross-validation designs.
In this study, we present a systematic empirical evaluation of standard
K-fold cross-validation and leave-source-out cross-validation methods in a
multi-source setting. We consider the task of electrocardiogram based
cardiovascular disease classification, combining and harmonizing the openly
available PhysioNet CinC Challenge 2021 and the Shandong Provincial Hospital
datasets for our study.
Our results show that K-fold cross-validation, both on single-source and
multi-source data, systemically overestimates prediction performance when the
end goal is to generalize to new sources. Leave-source-out cross-validation
provides more reliable performance estimates, having close to zero bias though
larger variability. The evaluation highlights the dangers of obtaining
misleading cross-validation results on medical data and demonstrates how these
issues can be mitigated when having access to multi-source data.

在这项研究中，我们对多源数据环境下的标准 K 折交叉验证和留源交叉验证方法进行了系统的实证评估，考虑了基于心电图的心血管疾病分类任务，并将 PhysioNet CinC Challenge 2021 和山东省医院的公开可用数据集进行了整合和协调。我们的结果表明，无论是单一源数据还是多源数据上的 K 折交叉验证都会在目标是推广到新的数据源时系统地高估预测性能。留源交叉验证提供了更可靠的性能估计，具有接近零的偏差但较大的可变性。这种评估凸显了关于在医学数据上获取具有误导性的交叉验证结果的危害，并展示了在拥有多源数据的情况下如何减轻这些问题。

临床机器学习中多源交叉验证的实证研究

Empirical investigation of multi-source cross-validation in clinical  machine learning

Developing clinical prediction models (e.g., mortality prediction) based on
electronic health records (EHRs) typically relies on expert opinion for feature
selection and adjusting observation window size. This burdens experts and
creates a bottleneck in the development process. We propose Retrieval-Enhanced
Medical prediction model (REMed) to address such challenges. REMed can
essentially evaluate an unlimited number of clinical events, select the
relevant ones, and make predictions. This approach effectively eliminates the
need for manual feature selection and enables an unrestricted observation
window. We verified these properties through experiments on 27 clinical tasks
and two independent cohorts from publicly available EHR datasets, where REMed
outperformed other contemporary architectures that aim to handle as many events
as possible. Notably, we found that the preferences of REMed align closely with
those of medical experts. We expect our approach to significantly expedite the
development of EHR prediction models by minimizing clinicians' need for manual
involvement.

通过检索增强型医学预测模型（REMed）在电子健康记录上构建临床预测模型，无需手动特征选择和受限观测窗口，验证实验显示 REMed 优于其他架构，能加速电子健康记录预测模型的开发过程。