We evaluate the uncertainty quality in neural networks using anomaly detection. We extract uncertainty measures (e.g. entropy) from the predictions of candidate models, use those measures as features for an anomaly detector, and gauge how well the detector differentiates known from unknown classes. We assign higher uncertainty quality to candidate models that lead to better detectors. We also propose a novel method for sampling a variational approximation of a Bayesian neural network, called One-Sample Bayesian Approximation (OSBA). We experiment on two datasets, MNIST and CIFAR10. We compare the following candidate neural network models: Maximum Likelihood, Bayesian Dropout, OSBA, and --- for MNIST --- the standard variational approximation. We show that Bayesian Dropout and OSBA provide better uncertainty information than Maximum Likelihood, and are essentially equivalent to the standard variational approximation, but much faster.

本文提出一种使用异常检测评估神经网络中不确定性质量的方法，通过从候选模型的预测中提取不确定性度量，将这些度量作为异常检测器的特征，并评估检测器区分已知类别和未知类别的能力。在MNIST和CIFAR10数据集上实验，比较了最大似然、贝叶斯随机失活、One-Sample贝叶斯逼近和标准变分逼近等候选神经网络模型的表现，结果表明贝叶斯随机失活和One-Sample贝叶斯逼近提供了更好的不确定性信息，并且在MNIST上与标准变分逼近相当，但速度更快。

已知未知：贝叶斯神经网络中的不确定质量