In this paper, we discuss feature engineering for single-pass uncertainty
estimation. For accurate uncertainty estimates, neural networks must extract
differences in the feature space that quantify uncertainty. This could be
achieved by current single-pass approaches that maintain feature distances
between data points as they traverse the network. While initial results are
promising, maintaining feature distances within the network representations
frequently inhibits information compression and opposes the learning objective.
We study this effect theoretically and empirically to arrive at a simple
conclusion: preserving feature distances in the output is beneficial when the
preserved features contribute to learning the label distribution and act in
opposition otherwise. We then propose Transitional Uncertainty with Layered
Intermediate Predictions (TULIP) as a simple approach to address the
shortcomings of current single-pass estimators. Specifically, we implement
feature preservation by extracting features from intermediate representations
before information is collapsed by subsequent layers. We refer to the
underlying preservation mechanism as transitional feature preservation. We show
that TULIP matches or outperforms current single-pass methods on standard
benchmarks and in practical settings where these methods are less reliable
(imbalances, complex architectures, medical modalities).

研究单次通过不确定性估计的特征工程，提出了一种名为 TULIP 的简单方法，通过在后续层折叠信息之前从中间表示中提取特征以实现特征保留，并在标准基准和实际环境中与当前的单次通过方法相匹配或优于其性能。