Deep neural networks (DNNs) have been successfully applied in various fields.
In DNNs, a large number of multiply-accumulate (MAC) operations is required to
be performed, posing critical challenges in applying them in
resource-constrained platforms, e.g., edge devices. Dynamic neural networks
have been introduced to allow a structural adaption, e.g., early-exit,
according to different inputs to reduce the computational cost of DNNs.
Existing early-exit techniques deploy classifiers at intermediate layers of
DNNs to push them to make a classification decision as early as possible.
However, the learned features at early layers might not be sufficient to
exclude all the irrelevant classes and decide the correct class, leading to
suboptimal results. To address this challenge, in this paper, we propose a
class-based early-exit for dynamic inference. Instead of pushing DNNs to make a
dynamic decision at intermediate layers, we take advantages of the learned
features in these layers to exclude as many irrelevant classes as possible, so
that later layers only have to determine the target class among the remaining
classes. Until at a layer only one class remains, this class is the
corresponding classification result. To realize this class-based exclusion, we
assign each class with a classifier at intermediate layers and train the
networks together with these classifiers. Afterwards, an exclusion strategy is
developed to exclude irrelevant classes at early layers. Experimental results
demonstrate the computational cost of DNNs in inference can be reduced
significantly.

该研究论文介绍了一种基于类别排除的动态推理方法，通过在中间层分配分类器并利用其学习到的特征排除尽可能多的无关类别，从而显著降低深度神经网络在推理中的计算成本。

神经网络动态推理的早期分类

Early Classification for Dynamic Inference of Neural Networks

Adaptive inference is a simple method for reducing inference costs. The
method works by maintaining multiple classifiers of different capacities, and
allocating resources to each test instance according to its difficulty. In this
work, we compare the two main approaches for adaptive inference, Early-Exit and
Multi-Model, when training data is limited. First, we observe that for models
with the same architecture and size, individual Multi-Model classifiers
outperform their Early-Exit counterparts by an average of 2.3%. We show that
this gap is caused by Early-Exit classifiers sharing model parameters during
training, resulting in conflicting gradient updates of model weights. We find
that despite this gap, Early-Exit still provides a better speed-accuracy
trade-off due to the overhead of the Multi-Model approach. To address these
issues, we propose SWEET (Separating Weights in Early Exit Transformers), an
Early-Exit fine-tuning method that assigns each classifier its own set of
unique model weights, not updated by other classifiers. We compare SWEET's
speed-accuracy curve to standard Early-Exit and Multi-Model baselines and find
that it outperforms both methods at fast speeds while maintaining comparable
scores to Early-Exit at slow speeds. Moreover, SWEET individual classifiers
outperform Early-Exit ones by 1.1% on average. SWEET enjoys the benefits of
both methods, paving the way for further reduction of inference costs in NLP.

本论文比较了有限训练数据情况下，自适应推断的两种主要方法：Early-Exit 和 Multi-Model，提出了一种名为 SWEET 的 Early-Exit 微调方法，在保持速度相当的情况下，优于 Early-Exit 和 Multi-Model，最终表明 SWEET 同时具备两种方法的优点，并为 NLP 中进一步降低推断成本打下了基础。

寻找甜点：低资源环境下自适应推理的分析与改进

Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference  in Low Resource Settings

Edge offloading for deep neural networks (DNNs) can be adaptive to the
input's complexity by using early-exit DNNs. These DNNs have side branches
throughout their architecture, allowing the inference to end earlier in the
edge. The branches estimate the accuracy for a given input. If this estimated
accuracy reaches a threshold, the inference ends on the edge. Otherwise, the
edge offloads the inference to the cloud to process the remaining DNN layers.
However, DNNs for image classification deals with distorted images, which
negatively impact the branches' estimated accuracy. Consequently, the edge
offloads more inferences to the cloud. This work introduces expert side
branches trained on a particular distortion type to improve robustness against
image distortion. The edge detects the distortion type and selects appropriate
expert branches to perform the inference. This approach increases the estimated
accuracy on the edge, improving the offloading decisions. We validate our
proposal in a realistic scenario, in which the edge offloads DNN inference to
Amazon EC2 instances.

通过专家侧支分支在边缘检测图像扭曲类型并根据该类型选择正确的侧支分支来执行推理的方法提高了边缘推理的估计精度，改善了边缘卸载决策。