Convolutional neural networks (CNNs) have recently become the
state-of-the-art in a diversity of AI tasks. Despite their popularity, CNN
inference still comes at a high computational cost. A growing body of work aims
to alleviate this by exploiting the difference in the classification difficulty
among samples and early-exiting at different stages of the network.
Nevertheless, existing studies on early exiting have primarily focused on the
training scheme, without considering the use-case requirements or the
deployment platform. This work presents HAPI, a novel methodology for
generating high-performance early-exit networks by co-optimising the placement
of intermediate exits together with the early-exit strategy at inference time.
Furthermore, we propose an efficient design space exploration algorithm which
enables the faster traversal of a large number of alternative architectures and
generates the highest-performing design, tailored to the use-case requirements
and target hardware. Quantitative evaluation shows that our system consistently
outperforms alternative search mechanisms and state-of-the-art early-exit
schemes across various latency budgets. Moreover, it pushes further the
performance of highly optimised hand-crafted early-exit CNNs, delivering up to
5.11x speedup over lightweight models on imposed latency-driven SLAs for
embedded devices.

本文提出了一种名为 HAPI 的新方法，通过在推理阶段与早期退出策略一起优化中间退出的放置位置，实现了高性能的早期退出网络生成。通过设计空间探索算法，我们提出了一种在各种延迟预算下始终优于其他搜索机制和最先进的早期退出方案的系统，它在强制嵌入式设备的延迟驱动 SLAs 的速度上提供了高效的方案。