Generalization theory has been established for sparse deep neural networks
under high-dimensional regime. Beyond generalization, parameter estimation is
also important since it is crucial for variable selection and interpretability
of deep neural networks. Current theoretical studies concerning parameter
estimation mainly focus on two-layer neural networks, which is due to the fact
that the convergence of parameter estimation heavily relies on the regularity
of the Hessian matrix, while the Hessian matrix of deep neural networks is
highly singular. To avoid the unidentifiability of deep neural networks in
parameter estimation, we propose to conduct nonparametric estimation of partial
derivatives with respect to inputs. We first show that model convergence of
sparse deep neural networks is guaranteed in that the sample complexity only
grows with the logarithm of the number of parameters or the input dimension
when the $\ell_{1}$-norm of parameters is well constrained. Then by bounding
the norm and the divergence of partial derivatives, we establish that the
convergence rate of nonparametric estimation of partial derivatives scales as
$\mathcal{O}(n^{-1/4})$, a rate which is slower than the model convergence rate
$\mathcal{O}(n^{-1/2})$. To the best of our knowledge, this study combines
nonparametric estimation and parametric sparse deep neural networks for the
first time. As nonparametric estimation of partial derivatives is of great
significance for nonlinear variable selection, the current results show the
promising future for the interpretability of deep neural networks.

非参数方法估计部分导数与稀疏深度神经网络的参数估计结合，为深度神经网络的可解释性提供了有前景的研究结果。

高维稀疏回归中的非参数估计稀疏深度神经网络

Sparse deep neural networks for nonparametric estimation in  high-dimensional sparse regression

Sparsity in Deep Neural Networks (DNNs) is studied extensively with the focus
of maximizing prediction accuracy given an overall parameter budget. Existing
methods rely on uniform or heuristic non-uniform sparsity budgets which have
sub-optimal layer-wise parameter allocation resulting in a) lower prediction
accuracy or b) higher inference cost (FLOPs). This work proposes Soft Threshold
Reparameterization (STR), a novel use of the soft-threshold operator on DNN
weights. STR smoothly induces sparsity while learning pruning thresholds
thereby obtaining a non-uniform sparsity budget. Our method achieves
state-of-the-art accuracy for unstructured sparsity in CNNs (ResNet50 and
MobileNetV1 on ImageNet-1K), and, additionally, learns non-uniform budgets that
empirically reduce the FLOPs by up to 50%. Notably, STR boosts the accuracy
over existing results by up to 10% in the ultra sparse (99%) regime and can
also be used to induce low-rank (structured sparsity) in RNNs. In short, STR is
a simple mechanism which learns effective sparsity budgets that contrast with
popular heuristics. Code, pretrained models and sparsity budgets are at
this https URL

本研究提出了一种基于软阈值重参数化（STR）的方法，该方法通过学习修剪阈值实现平滑稀疏（非均匀稀疏），从而获得非均匀稀疏预算，进而在 CNNs 中实现非结构化稀疏，在 ImageNet-1K 上实现了最先进的准确性，并且可以用于在 RNNs 中引出低秩（结构稀疏）。