The success of DNN pruning has led to the development of energy-efficient
inference accelerators that support pruned models with sparse weight and
activation tensors. Because the memory layouts and dataflows in these
architectures are optimized for the access patterns during
$\mathit{inference}$, however, they do not efficiently support the emerging
sparse $\mathit{training}$ techniques.
In this paper, we demonstrate (a) that accelerating sparse training requires
a co-design approach where algorithms are adapted to suit the constraints of
hardware, and (b) that hardware for sparse DNN training must tackle constraints
that do not arise in inference accelerators. As proof of concept, we adapt a
sparse training algorithm to be amenable to hardware acceleration; we then
develop dataflow, data layout, and load-balancing techniques to accelerate it.
The resulting system is a sparse DNN training accelerator that produces
pruned models with the same accuracy as dense models without first training,
then pruning, and finally retraining, a dense model. Compared to training the
equivalent unpruned models using a state-of-the-art DNN accelerator without
sparse training support, Procrustes consumes up to 3.26$\times$ less energy and
offers up to 4$\times$ speedup across a range of models, while pruning weights
by an order of magnitude and maintaining unpruned accuracy.

本文研究了适用于稀疏权重和激活张量的能源高效型深度神经网络 (DNN) 修剪算法，并提出了一种稀疏 DNN 训练加速器，可以在不牺牲精度的情况下消耗更少的能源和获得更快的速度。该研究表明，加速稀疏训练需要一种协同设计方法，其中算法被适应到硬件的限制，并且与推断加速器不同，稀疏 DNN 训练硬件必须应对不同的限制。