Deep reinforcement learning(DRL) has shown significant promise in a wide
range of applications including computer games and robotics. Yet, training DRL
policies consume extraordinary computing resources resulting in dense policies
which are prone to overfitting. Moreover, inference with dense DRL policies
limit their practical applications, especially in edge computing. Techniques
such as pruning and singular value decomposition have been used with deep
learning models to achieve sparsification and model compression to limit
overfitting and reduce memory consumption. However, these techniques resulted
in sub-optimal performance with notable decay in rewards. $L_1$ and $L_2$
regularization techniques have been proposed for neural network sparsification
and sparse auto-encoder development, but their implementation in DRL
environments has not been apparent. We propose a novel
$L_0$-norm-regularization technique using an optimal sparsity map to sparsify
DRL policies and promote their decomposition to a lower rank without decay in
rewards. We evaluated our $L_0$-norm-regularization technique across five
different environments (Cartpole-v1, Acrobat-v1, LunarLander-v2,
SuperMarioBros-7.1.v0 and Surgical Robot Learning) using several on-policy and
off-policy algorithms. We demonstrated that the $L_0$-norm-regularized DRL
policy in the SuperMarioBros environment achieved 93% sparsity and gained 70%
compression when subjected to low-rank decomposition, while significantly
outperforming the dense policy. Additionally, the $L_0$-norm-regularized DRL
policy in the Surgical Robot Learning environment achieved a 36% sparsification
and gained 46% compression when decomposed to a lower rank, while being
performant. The results suggest that our custom $L_0$-norm-regularization
technique for sparsification of DRL policies is a promising avenue to reduce
computational resources and limit overfitting.

使用 $L_0$- 范数正则化技术稀疏化深度强化学习策略，通过低秩分解提供压缩效果，以减少计算资源消耗和限制过拟合。