Machine learning algorithms have grown in sophistication over the years and
are increasingly deployed for real-life applications. However, when using
machine learning techniques in practical settings, particularly in high-risk
applications such as medicine and engineering, obtaining the failure
probability of the predictive model is critical. We refer to this problem as
the risk-assessment task. We focus on regression algorithms and the
risk-assessment task of computing the probability of the true label lying
inside an interval defined around the model's prediction. We solve the
risk-assessment problem using the conformal prediction approach, which provides
prediction intervals that are guaranteed to contain the true label with a given
probability. Using this coverage property, we prove that our approximated
failure probability is conservative in the sense that it is not lower than the
true failure probability of the ML algorithm. We conduct extensive experiments
to empirically study the accuracy of the proposed method for problems with and
without covariate shift. Our analysis focuses on different modeling regimes,
dataset sizes, and conformal prediction methodologies.

机器学习算法在实际应用中发展日益复杂，特别是在医学和工程等高风险应用中使用机器学习技术时，预测模型的失败概率至关重要。我们提出了风险评估任务，并侧重于回归算法和计算模型预测的真实标签在一个定义好的区间内的概率。我们通过使用符合预测方法来解决风险评估问题，该方法提供了一定概率内包含真实标签的预测区间。通过该覆盖性质，我们证明了所提方法的近似失败概率是保守的，不低于 ML 算法的真实失败概率。我们进行了大量实验证明了所提方法在存在和不存在协变量转移的问题中的准确性，并重点研究了不同建模方案、数据集大小和符合预测方法学。