Sparse neural networks are highly desirable in deep learning in reducing its complexity. The goal of this paper is to study how choices of regularization parameters influence the sparsity level of learned neural networks. We first derive the $\ell_1$-norm sparsity-promoting deep learning models including single and multiple regularization parameters models, from a statistical viewpoint. We then characterize the sparsity level of a regularized neural network in terms of the choice of the regularization parameters. Based on the characterizations, we develop iterative algorithms for selecting regularization parameters so that the weight parameters of the resulting deep neural network enjoy prescribed sparsity levels. Numerical experiments are presented to demonstrate the effectiveness of the proposed algorithms in choosing desirable regularization parameters and obtaining corresponding neural networks having both of predetermined sparsity levels and satisfactory approximation accuracy.

本研究关注于稀疏神经网络的重要性，探讨正则化参数的选择如何影响学习到的神经网络的稀疏程度。通过从统计学角度导出带有$\ell_1$-范数的稀疏促进深度学习模型，本文发展了选择正则化参数的迭代算法，以实现预定的稀疏水平，并在数值实验中验证了方法的有效性。

带有$\ell_1$正则化的稀疏深度学习模型