While improving prediction accuracy has been the focus of machine learning in
recent years, this alone does not suffice for reliable decision-making.
Deploying learning systems in consequential settings also requires calibrating
and communicating the uncertainty of predictions. To convey instance-wise
uncertainty for prediction tasks, we show how to generate set-valued
predictions from a black-box predictor that control the expected loss on future
test points at a user-specified level. Our approach provides explicit
finite-sample guarantees for any dataset by using a holdout set to calibrate
the size of the prediction sets. This framework enables simple,
distribution-free, rigorous error control for many tasks, and we demonstrate it
in five large-scale machine learning problems: (1) classification problems
where some mistakes are more costly than others; (2) multi-label
classification, where each observation has multiple associated labels; (3)
classification problems where the labels have a hierarchical structure; (4)
image segmentation, where we wish to predict a set of pixels containing an
object of interest; and (5) protein structure prediction. Lastly, we discuss
extensions to uncertainty quantification for ranking, metric learning and
distributionally robust learning.

本文介绍了一种基于黑盒预测器实现集值预测、控制用户指定水平下的未来测试点期望损失大小以及使用 holdout 集来校准优化预测集大小的方法，这种方法能够提供简单、分布自由和严密的误差控制，适用于多个任务，如分类问题、多标记分类、分类问题中带有层次结构的标签、图像分割和蛋白质结构预测，还有拓展探讨。