Current dynamic networks and dynamic pruning methods have shown their
promising capability in reducing theoretical computation complexity. However,
dynamic sparse patterns on convolutional filters fail to achieve actual
acceleration in real-world implementation, due to the extra burden of indexing,
weight-copying, or zero-masking. Here, we explore a dynamic network slimming
regime, named Dynamic Slimmable Network (DS-Net), which aims to achieve good
hardware-efficiency via dynamically adjusting filter numbers of networks at
test time with respect to different inputs, while keeping filters stored
statically and contiguously in hardware to prevent the extra burden. Our DS-Net
is empowered with the ability of dynamic inference by the proposed
double-headed dynamic gate that comprises an attention head and a slimming head
to predictively adjust network width with negligible extra computation cost. To
ensure generality of each candidate architecture and the fairness of gate, we
propose a disentangled two-stage training scheme inspired by one-shot NAS. In
the first stage, a novel training technique for weight-sharing networks named
In-place Ensemble Bootstrapping is proposed to improve the supernet training
efficacy. In the second stage, Sandwich Gate Sparsification is proposed to
assist the gate training by identifying easy and hard samples in an online way.
Extensive experiments demonstrate our DS-Net consistently outperforms its
static counterparts as well as state-of-the-art static and dynamic model
compression methods by a large margin (up to 5.9%). Typically, DS-Net achieves
2-4x computation reduction and 1.62x real-world acceleration over ResNet-50 and
MobileNet with minimal accuracy drops on ImageNet. Code release:
this https URL .

本文提出了一种名为 DS-Net 的动态可整合网络概念，它通过在测试时动态调整网络的卷积滤波器数量，从而实现好的硬件效率。同时，DS-Net 通过一种双头动态门的机制，具有动态推理的能力，并使用一种两阶段训练方案进行训练。实验结果表明，DS-Net 在 ImageNet 数据集上与 ResNet-50 和 MobileNet 模型相比，具有更高的计算性能和实际加速性能。