Early Exit Neural Networks (EENNs) present a solution to enhance the
efficiency of neural network deployments. However, creating EENNs is
challenging and requires specialized domain knowledge, due to the large amount
of additional design choices. To address this issue, we propose an automated
augmentation flow that focuses on converting an existing model into an EENN. It
performs all required design decisions for the deployment to heterogeneous or
distributed hardware targets: Our framework constructs the EENN architecture,
maps its subgraphs to the hardware targets, and configures its decision
mechanism. To the best of our knowledge, it is the first framework that is able
to perform all of these steps.
We evaluated our approach on a collection of Internet-of-Things and standard
image classification use cases. For a speech command detection task, our
solution was able to reduce the mean operations per inference by 59.67%. For an
ECG classification task, it was able to terminate all samples early, reducing
the mean inference energy by 74.9% and computations by 78.3%. On CIFAR-10, our
solution was able to achieve up to a 58.75% reduction in computations.
The search on a ResNet-152 base model for CIFAR-10 took less than nine hours
on a laptop CPU. Our proposed approach enables the creation of EENN optimized
for IoT environments and can reduce the inference cost of Deep Learning
applications on embedded and fog platforms, while also significantly reducing
the search cost - making it more accessible for scientists and engineers in
industry and research. The low search cost improves the accessibility of EENNs,
with the potential to improve the efficiency of neural networks in a wide range
of practical applications.

我们提出了一种自动增强流程，将现有模型转换为 EENN，并为异构或分布式硬件目标执行所有必需的设计决策。我们的方法在物联网和标准图像分类用例中都能有效降低推理成本，并大大减少 EENNs 的搜索成本，提高了神经网络在各种实际应用中的效率。