We propose to execute deep neural networks (DNNs) with dynamic and sparse
graph (DSG) structure for compressive memory and accelerative execution during
both training and inference. The great success of DNNs motivates the pursuing
of lightweight models for the deployment onto embedded devices. However, most
of the previous studies optimize for inference while neglect training or even
complicate it. Training is far more intractable, since (i) the neurons dominate
the memory cost rather than the weights in inference; (ii) the dynamic
activation makes previous sparse acceleration via one-off optimization on fixed
weight invalid; (iii) batch normalization (BN) is critical for maintaining
accuracy while its activation reorganization damages the sparsity. To address
these issues, DSG activates only a small amount of neurons with high
selectivity at each iteration via a dimension-reduction search (DRS) and
obtains the BN compatibility via a double-mask selection (DMS). Experiments
show significant memory saving (1.7-4.5x) and operation reduction (2.3-4.4x)
with little accuracy loss on various benchmarks.

本文提出了一种通过动态和稀疏的图结构来执行深度神经网络（DNNs）进行压缩存储和加速执行的方法，该方法在训练和推断时均可应用，以优化深度神经网络的轻量化模型。实验证明该方法可以显著提供存储和操作的性能，并在各种基准测试中几乎无损失地维护准确性。