Sparse deep learning aims to address the challenge of huge storage consumption by deep neural networks, and to recover the sparse structure of target functions. Although tremendous empirical successes have been achieved, most sparse deep learning algorithms are lacking of theoretical support. On the other hand, another line of works have proposed theoretical frameworks that are computationally infeasible. In this paper, we train sparse deep neural networks with a fully Bayesian treatment under spike-and-slab priors, and develop a set of computationally efficient variational inferences via continuous relaxation of Bernoulli distribution. The variational posterior contraction rate is provided, which justifies the consistency of the proposed variational Bayes method. Notably, our empirical results demonstrate that this variational procedure provides uncertainty quantification in terms of Bayesian predictive distribution and is also capable to accomplish consistent variable selection by training a sparse multi-layer neural network.

本文旨在通过完全贝叶斯处理下的尖峰-平板先验训练稀疏深度神经网络，通过连续放松伯努利分布开发一组计算有效的变分推断方法。实证结果表明，这种变分程序不仅提供了关于贝叶斯预测分布的不确定性量化，而且还能通过训练稀疏多层神经网络实现一致的变量选择。

稀疏深度学习的高效变分推断及理论保证