By leveraging the data sample diversity, the early-exit network recently
emerges as a prominent neural network architecture to accelerate the deep
learning inference process. However, intermediate classifiers of the early
exits introduce additional computation overhead, which is unfavorable for
resource-constrained edge artificial intelligence (AI). In this paper, we
propose an early exit prediction mechanism to reduce the on-device computation
overhead in a device-edge co-inference system supported by early-exit networks.
Specifically, we design a low-complexity module, namely the Exit Predictor, to
guide some distinctly "hard" samples to bypass the computation of the early
exits. Besides, considering the varying communication bandwidth, we extend the
early exit prediction mechanism for latency-aware edge inference, which adapts
the prediction thresholds of the Exit Predictor and the confidence thresholds
of the early-exit network via a few simple regression models. Extensive
experiment results demonstrate the effectiveness of the Exit Predictor in
achieving a better tradeoff between accuracy and on-device computation overhead
for early-exit networks. Besides, compared with the baseline methods, the
proposed method for latency-aware edge inference attains higher inference
accuracy under different bandwidth conditions.

本文提出了一种早期预测机制 'Exit Predictor'，通过引导某些明显 “困难” 的数据样本绕过早期出口的计算，从而降低设备边缘混合推理系统中早期退出网络的设备计算负担。同时，还考虑了通信带宽的变化，在延迟感知的边缘推理中通过几个简单的回归模型来调整 Exit Predictor 的预测阈值和早期退出网络的置信度阈值，以适应不同带宽条件下推理的需求。实验结果表明，Exit Predictor 对于早期退出网络在准确性和设备计算负担之间取得更好的折衷效果。与基线方法相比，在不同带宽条件下，提出的延迟感知的边缘推理方法具有更高的推理准确性。