Deep Learning is becoming increasingly relevant in Embedded and
Internet-of-things applications. However, deploying models on embedded devices
poses a challenge due to their resource limitations. This can impact the
model's inference accuracy and latency. One potential solution are Early Exit
Neural Networks, which adjust model depth dynamically through additional
classifiers attached between their hidden layers. However, the real-time
termination decision mechanism is critical for the system's efficiency,
latency, and sustained accuracy.
This paper introduces Difference Detection and Temporal Patience as decision
mechanisms for Early Exit Neural Networks. They leverage the temporal
correlation present in sensor data streams to efficiently terminate the
inference. We evaluate their effectiveness in health monitoring, image
classification, and wake-word detection tasks. Our novel contributions were
able to reduce the computational footprint compared to established decision
mechanisms significantly while maintaining higher accuracy scores. We achieved
a reduction of mean operations per inference by up to 80% while maintaining
accuracy levels within 5% of the original model.
These findings highlight the importance of considering temporal correlation
in sensor data to improve the termination decision.

利用差异检测和时间耐心作为早期中止神经网络的决策机制，可以显著降低计算资源占用，同时保持较高的准确率。这些研究结果强调了考虑传感器数据中的时间相关性以改进终止决策的重要性。