Bayesian neural networks (BNN) and deep ensembles are principled approaches
to estimate the predictive uncertainty of a deep learning model. However their
practicality in real-time, industrial-scale applications are limited due to
their heavy memory and inference cost. This motivates us to study principled
approaches to high-quality uncertainty estimation that require only a single
deep neural network (DNN). By formalizing the uncertainty quantification as a
minimax learning problem, we first identify input distance awareness, i.e., the
model's ability to quantify the distance of a testing example from the training
data in the input space, as a necessary condition for a DNN to achieve
high-quality (i.e., minimax optimal) uncertainty estimation. We then propose
Spectral-normalized Neural Gaussian Process (SNGP), a simple method that
improves the distance-awareness ability of modern DNNs, by adding a weight
normalization step during training and replacing the output layer with a
Gaussian process. On a suite of vision and language understanding tasks and on
modern architectures (Wide-ResNet and BERT), SNGP is competitive with deep
ensembles in prediction, calibration and out-of-domain detection, and
outperforms the other single-model approaches.

SNGP 基于权重规范化和高斯过程，解决了深度神经网络中单模型在预测、校准和域外检测上的不确定性问题。