Large pretrained models, coupled with fine-tuning, are slowly becoming
established as the dominant architecture in machine learning. Even though these
models offer impressive performance, their practical application is often
limited by the prohibitive amount of resources required for every inference.
Early-exiting dynamic neural networks (EDNN) circumvent this issue by allowing
a model to make some of its predictions from intermediate layers (i.e.,
early-exit). Training an EDNN architecture is challenging as it consists of two
intertwined components: the gating mechanism (GM) that controls early-exiting
decisions and the intermediate inference modules (IMs) that perform inference
from intermediate representations. As a result, most existing approaches rely
on thresholding confidence metrics for the gating mechanism and strive to
improve the underlying backbone network and the inference modules. Although
successful, this approach has two fundamental shortcomings: 1) the GMs and the
IMs are decoupled during training, leading to a train-test mismatch; and 2) the
thresholding gating mechanism introduces a positive bias into the predictive
probabilities, making it difficult to readily extract uncertainty information.
We propose a novel architecture that connects these two modules. This leads to
significant performance improvements on classification datasets and enables
better uncertainty characterization capabilities.

大型预训练模型与微调相结合，在机器学习中逐渐成为主要的架构。早期退出动态神经网络（EDNN）通过允许模型从中间层进行预测（即早期退出），绕过了每个推理所需的资源过多的限制。我们提出了一种连接这两个模块的新型架构，这在分类数据集上带来了显著的性能提升，并且具有更好的不确定性特征。