Sparse training has received an upsurging interest in machine learning due to
its tantalizing saving potential for the entire training process as well as
inference. Dynamic sparse training (DST), as a leading sparse training
approach, can train deep neural networks at high sparsity from scratch to match
the performance of their dense counterparts. However, most if not all DST prior
arts demonstrate their effectiveness on unstructured sparsity with highly
irregular sparse patterns, which receives limited support in common hardware.
This limitation hinders the usage of DST in practice. In this paper, we propose
Channel-aware dynamic sparse (Chase), which for the first time seamlessly
translates the promise of unstructured dynamic sparsity to GPU-friendly
channel-level sparsity (not fine-grained N:M or group sparsity) during one
end-to-end training process, without any ad-hoc operations. The resulting small
sparse networks can be directly accelerated by commodity hardware, without
using any particularly sparsity-aware hardware accelerators. This appealing
outcome is partially motivated by a hidden phenomenon of dynamic sparsity:
off-the-shelf unstructured DST implicitly involves biased parameter
reallocation across channels, with a large fraction of channels (up to 60\%)
being sparser than others. By progressively identifying and removing these
channels during training, our approach translates unstructured sparsity to
channel-wise sparsity. Our experimental results demonstrate that Chase achieves
1.7 X inference throughput speedup on common GPU devices without compromising
accuracy with ResNet-50 on ImageNet. We release our codes in
this https URL

本文提出 Channel-aware dynamic sparse (Chase) 方法：将 unstructured dynamic sparsity 转变为 GPU-friendly channel-level sparsity 加速 inference，通过逐渐去除 biased parameter reallocation across channels，不损失准确率地实现了 1.7 X inference throughput speedup on common GPU devices with ResNet-50 on ImageNet。