Bayesian deep learning seeks to equip deep neural networks with the ability
to precisely quantify their predictive uncertainty, and has promised to make
deep learning more reliable for safety-critical real-world applications. Yet,
existing Bayesian deep learning methods fall short of this promise; new methods
continue to be evaluated on unrealistic test beds that do not reflect the
complexities of downstream real-world tasks that would benefit most from
reliable uncertainty quantification. We propose the RETINA Benchmark, a set of
real-world tasks that accurately reflect such complexities and are designed to
assess the reliability of predictive models in safety-critical scenarios.
Specifically, we curate two publicly available datasets of high-resolution
human retina images exhibiting varying degrees of diabetic retinopathy, a
medical condition that can lead to blindness, and use them to design a suite of
automated diagnosis tasks that require reliable predictive uncertainty
quantification. We use these tasks to benchmark well-established and
state-of-the-art Bayesian deep learning methods on task-specific evaluation
metrics. We provide an easy-to-use codebase for fast and easy benchmarking
following reproducibility and software design principles. We provide
implementations of all methods included in the benchmark as well as results
computed over 100 TPU days, 20 GPU days, 400 hyperparameter configurations, and
evaluation on at least 6 random seeds each.

本文提出了 RETINA 基准测试，用于评估贝叶斯深度学习方法的可靠性和在安全关键场景下的预测不确定性量化。作者使用两个公开数据集设计了一组自动诊断任务，要求对高分辨率人类视网膜图像进行可靠的预测不确定性量化，以此来考察现有的贝叶斯深度学习方法在现实世界任务中的表现。实现结果表明，所提出的 RETINA 基准测试是可行的，有望在配备可靠的预测不确定性工具的深度学习应用中发挥重要作用。