Dynamic computation has emerged as a promising avenue to enhance the
inference efficiency of deep networks. It allows selective activation of
computational units, leading to a reduction in unnecessary computations for
each input sample. However, the actual efficiency of these dynamic models can
deviate from theoretical predictions. This mismatch arises from: 1) the lack of
a unified approach due to fragmented research; 2) the focus on algorithm design
over critical scheduling strategies, especially in CUDA-enabled GPU contexts;
and 3) challenges in measuring practical latency, given that most libraries
cater to static operations. Addressing these issues, we unveil the
Latency-Aware Unified Dynamic Networks (LAUDNet), a framework that integrates
three primary dynamic paradigms-spatially adaptive computation, dynamic layer
skipping, and dynamic channel skipping. To bridge the theoretical and practical
efficiency gap, LAUDNet merges algorithmic design with scheduling optimization,
guided by a latency predictor that accurately gauges dynamic operator latency.
We've tested LAUDNet across multiple vision tasks, demonstrating its capacity
to notably reduce the latency of models like ResNet-101 by over 50% on
platforms such as V100, RTX3090, and TX2 GPUs. Notably, LAUDNet stands out in
balancing accuracy and efficiency. Code is available at:
this https URL

动态计算已成为提高深度网络推理效率的一种有前途的方法，我们提出了一个名为 “LAUDNet” 的框架，它集成了三个主要的动态范例，以缩短模型的延迟，通过算法设计与调度优化相结合，准确衡量动态操作延迟的潜在预测器指导。我们在多个视觉任务上测试了 LAUDNet，其在 V100、RTX3090 和 TX2 GPU 上能够显著降低 ResNet-101 等模型的延迟超过 50％，而且在准确性和效率之间取得良好的平衡。