Recent work has shown that deep reinforcement learning agents have difficulty
in effectively using their network parameters. We leverage prior insights into
the advantages of sparse training techniques and demonstrate that gradual
magnitude pruning enables agents to maximize parameter effectiveness. This
results in networks that yield dramatic performance improvements over
traditional networks and exhibit a type of "scaling law", using only a small
fraction of the full network parameters.

近期的研究表明，深度强化学习代理在有效利用网络参数方面存在困难。我们利用先前对稀疏训练技术优势的了解，证明逐渐减少参数的大小可以使代理最大化参数的有效性。这导致网络在性能上比传统网络有显著的改进，并呈现出一种 “按比例尺” 的特性，只使用了网络参数的一小部分。

深度强化学习中的剪枝网络是良好网络

In deep reinforcement learning, a pruned network is a good network

The use of sparse neural networks has seen rapid growth in recent years,
particularly in computer vision. Their appeal stems largely from the reduced
number of parameters required to train and store, as well as in an increase in
learning efficiency. Somewhat surprisingly, there have been very few efforts
exploring their use in Deep Reinforcement Learning (DRL). In this work we
perform a systematic investigation into applying a number of existing sparse
training techniques on a variety of DRL agents and environments. Our results
corroborate the findings from sparse training in the computer vision domain -
sparse networks perform better than dense networks for the same parameter count
- in the DRL domain. We provide detailed analyses on how the various components
in DRL are affected by the use of sparse networks and conclude by suggesting
promising avenues for improving the effectiveness of sparse training methods,
as well as for advancing their use in DRL.

在深度强化学习领域，这项工作系统地研究了应用多种现有的稀疏训练技术在各种强化学习代理和环境中的可行性，结果发现，稀疏网络比密集网络在相同数量参数下表现更好，我们提供了有关如何改善稀疏训练方法有效性以及推进其在深度强化学习中应用的有益思路。

深度强化学习稀疏训练的现状

The State of Sparse Training in Deep Reinforcement Learning

The success of DNN pruning has led to the development of energy-efficient
inference accelerators that support pruned models with sparse weight and
activation tensors. Because the memory layouts and dataflows in these
architectures are optimized for the access patterns during
$\mathit{inference}$, however, they do not efficiently support the emerging
sparse $\mathit{training}$ techniques.
In this paper, we demonstrate (a) that accelerating sparse training requires
a co-design approach where algorithms are adapted to suit the constraints of
hardware, and (b) that hardware for sparse DNN training must tackle constraints
that do not arise in inference accelerators. As proof of concept, we adapt a
sparse training algorithm to be amenable to hardware acceleration; we then
develop dataflow, data layout, and load-balancing techniques to accelerate it.
The resulting system is a sparse DNN training accelerator that produces
pruned models with the same accuracy as dense models without first training,
then pruning, and finally retraining, a dense model. Compared to training the
equivalent unpruned models using a state-of-the-art DNN accelerator without
sparse training support, Procrustes consumes up to 3.26$\times$ less energy and
offers up to 4$\times$ speedup across a range of models, while pruning weights
by an order of magnitude and maintaining unpruned accuracy.

本文研究了适用于稀疏权重和激活张量的能源高效型深度神经网络 (DNN) 修剪算法，并提出了一种稀疏 DNN 训练加速器，可以在不牺牲精度的情况下消耗更少的能源和获得更快的速度。该研究表明，加速稀疏训练需要一种协同设计方法，其中算法被适应到硬件的限制，并且与推断加速器不同，稀疏 DNN 训练硬件必须应对不同的限制。