Efficiently and reliably estimating uncertainty is an important objective in
deep learning. It is especially pertinent to autoregressive sequence tasks,
where training and inference costs are typically very high. However, existing
research has predominantly focused on tasks with static data such as image
classification. In this work, we investigate Ensemble Distribution Distillation
(EDD) applied to large-scale natural language sequence-to-sequence data. EDD
aims to compress the superior uncertainty performance of an expensive (teacher)
ensemble into a cheaper (student) single model. Importantly, the ability to
separate knowledge (epistemic) and data (aleatoric) uncertainty is retained.
Existing probability-space approaches to EDD, however, are difficult to scale
to large vocabularies. We show, for modern transformer architectures on
large-scale translation tasks, that modelling the ensemble logits, instead of
softmax probabilities, leads to significantly better students. Moreover, the
students surprisingly even outperform Deep Ensembles by up to ~10% AUROC on
out-of-distribution detection, whilst matching them at in-distribution
translation.

本研究考察了在大规模自然语言序列到序列数据中应用 Ensemble Distribution Distillation （EDD）来压缩昂贵的集成模型的优越性能为成本较低的单一模型的有效性，既能够分离知识（认识不一致性）和数据（随机数据）不确定性，还能解决大词汇量数据的复杂性，并在非分布式检测中表现出色。

基于逻辑回归的集成分布蒸馏用于鲁棒性自回归序列不确定性

Logit-Based Ensemble Distribution Distillation for Robust Autoregressive  Sequence Uncertainties

Ensembles of machine learning models yield improved system performance as
well as robust and interpretable uncertainty estimates; however, their
inference costs may often be prohibitively high. \emph{Ensemble Distribution
Distillation} is an approach that allows a single model to efficiently capture
both the predictive performance and uncertainty estimates of an ensemble. For
classification, this is achieved by training a Dirichlet distribution over the
ensemble members' output distributions via the maximum likelihood criterion.
Although theoretically principled, this criterion exhibits poor convergence
when applied to large-scale tasks where the number of classes is very high. In
our work, we analyze this effect and show that the Dirichlet log-likelihood
criterion classes with low probability induce larger gradients than
high-probability classes. This forces the model to focus on the distribution of
the ensemble tail-class probabilities. We propose a new training objective that
minimizes the reverse KL-divergence to a \emph{Proxy-Dirichlet} target derived
from the ensemble. This loss resolves the gradient issues of Ensemble
Distribution Distillation, as we demonstrate both theoretically and empirically
on the ImageNet and WMT17 En-De datasets containing 1000 and 40,000 classes,
respectively.

这篇论文提出了一种称为 “Ensemble Distribution Distillation” 的机器学习方法，通过训练 Dirichlet 分布来捕捉整个集成预测和不确定性估计。为了解决在大规模任务中 Dirichlet log-likelihood 的梯度问题，作者提出了一种新的训练目标来最小化 Ensemble Distribution Distillation 的反向 KL 散度，从而在 ImageNet 和 WMT17 En-De 数据集上展示了它的优越性。

使用代理目标扩展集成分布蒸馏到多类问题

Scaling Ensemble Distribution Distillation to Many Classes with Proxy  Targets

Prior Networks are a recently developed class of models which yield
interpretable measures of uncertainty and have been shown to outperform
state-of-the-art ensemble approaches on a range of tasks. They can also be used
to distill an ensemble of models via Ensemble Distribution Distillation
(EnD$^2$), such that its accuracy, calibration and uncertainty estimates are
retained within a single model. However, Prior Networks have so far been
developed only for classification tasks. This work extends Prior Networks and
EnD$^2$ to regression tasks by considering the Normal-Wishart distribution. The
properties of Regression Prior Networks are demonstrated on synthetic data,
selected UCI datasets and a monocular depth estimation task, where they yield
performance competitive with ensemble approaches.

本文延伸了 Prior Networks 和 EnD$^2$ 模型，通过正态 - 旺季分布，将其应用于回归任务，实现模型的可解释性、不确定性而保留高精度，相关实验结果表明该模型在各领域均表现出与集成模型相当的性能。

回归先验网络

Regression Prior Networks

Ensembles of models often yield improvements in system performance. These
ensemble approaches have also been empirically shown to yield robust measures
of uncertainty, and are capable of distinguishing between different
\emph{forms} of uncertainty. However, ensembles come at a computational and
memory cost which may be prohibitive for many applications. There has been
significant work done on the distillation of an ensemble into a single model.
Such approaches decrease computational cost and allow a single model to achieve
an accuracy comparable to that of an ensemble. However, information about the
\emph{diversity} of the ensemble, which can yield estimates of different forms
of uncertainty, is lost. This work considers the novel task of \emph{Ensemble
Distribution Distillation} (EnD$^2$) --- distilling the distribution of the
predictions from an ensemble, rather than just the average prediction, into a
single model. EnD$^2$ enables a single model to retain both the improved
classification performance of ensemble distillation as well as information
about the diversity of the ensemble, which is useful for uncertainty
estimation. A solution for EnD$^2$ based on Prior Networks, a class of models
which allow a single neural network to explicitly model a distribution over
output distributions, is proposed in this work. The properties of EnD$^2$ are
investigated on both an artificial dataset, and on the CIFAR-10, CIFAR-100 and
TinyImageNet datasets, where it is shown that EnD$^2$ can approach the
classification performance of an ensemble, and outperforms both standard DNNs
and Ensemble Distillation on the tasks of misclassification and
out-of-distribution input detection.

该论文提出了一种名为 EnD^2 的新方法，它使用 Prior Networks 对模型的输出分布进行建模，并使单个模型保留了集成模型减少计算成本和提高分类性能的同时，还能保留集合多样性的信息，用于估计不确定性。该方法不仅可适用于人工数据集，还取得了在 CIFAR-10、CIFAR-100 和 TinyImageNet 数据集上优于其他方法的成果。