Accurate uncertainty quantification is a major challenge in deep learning, as
neural networks can make overconfident errors and assign high confidence
predictions to out-of-distribution (OOD) inputs. The most popular approaches to
estimate predictive uncertainty in deep learning are methods that combine
predictions from multiple neural networks, such as Bayesian neural networks
(BNNs) and deep ensembles. However their practicality in real-time,
industrial-scale applications are limited due to the high memory and
computational cost. Furthermore, ensembles and BNNs do not necessarily fix all
the issues with the underlying member networks. In this work, we study
principled approaches to improve uncertainty property of a single network,
based on a single, deterministic representation. By formalizing the uncertainty
quantification as a minimax learning problem, we first identify distance
awareness, i.e., the model's ability to quantify the distance of a testing
example from the training data, as a necessary condition for a DNN to achieve
high-quality (i.e., minimax optimal) uncertainty estimation. We then propose
Spectral-normalized Neural Gaussian Process (SNGP), a simple method that
improves the distance-awareness ability of modern DNNs with two simple changes:
(1) applying spectral normalization to hidden weights to enforce bi-Lipschitz
smoothness in representations and (2) replacing the last output layer with a
Gaussian process layer. On a suite of vision and language understanding
benchmarks, SNGP outperforms other single-model approaches in prediction,
calibration and out-of-domain detection. Furthermore, SNGP provides
complementary benefits to popular techniques such as deep ensembles and data
augmentation, making it a simple and scalable building block for probabilistic
deep learning. Code is open-sourced at
this https URL

研究了如何改进深度神经网络的不确定性量化能力，基于确定性表示将不确定性量化形式化为极小化 - 极大化学习问题，提出了一种名为 SNGP 的方法，通过两个简单的改变来提高现代 DNN 的距离感知能力，取得了较好的预测、校准和超领域检测性能。

距离感知简单方法改进单模型深度不确定性

A Simple Approach to Improve Single-Model Deep Uncertainty via  Distance-Awareness

The success of minimax learning problems of generative adversarial networks
(GANs) has been observed to depend on the minimax optimization algorithm used
for their training. This dependence is commonly attributed to the convergence
speed and robustness properties of the underlying optimization algorithm. In
this paper, we show that the optimization algorithm also plays a key role in
the generalization performance of the trained minimax model. To this end, we
analyze the generalization properties of standard gradient descent ascent (GDA)
and proximal point method (PPM) algorithms through the lens of algorithmic
stability under both convex concave and non-convex non-concave minimax
settings. While the GDA algorithm is not guaranteed to have a vanishing excess
risk in convex concave problems, we show the PPM algorithm enjoys a bounded
excess risk in the same setup. For non-convex non-concave problems, we compare
the generalization performance of stochastic GDA and GDmax algorithms where the
latter fully solves the maximization subproblem at every iteration. Our
generalization analysis suggests the superiority of GDA provided that the
minimization and maximization subproblems are solved simultaneously with
similar learning rates. We discuss several numerical results indicating the
role of optimization algorithms in the generalization of the learned minimax
models.

本文研究发现优化算法在训练最大 - 最小学习问题的生成式对抗网络中发挥了关键作用，涉及泛化性能和算法稳定性等方面，而梯度下降上升算法则是其中一种表现优越的算法。