Spike-and-Slab Deep Learning (SS-DL) is a fully Bayesian alternative to Dropout for improving generalizability of deep ReLU networks. This new type of regularization enables provable recovery of smooth input-output maps with unknown levels of smoothness. Indeed, we show that the posterior distribution concentrates at the near minimax rate for $\alpha$-H\"older smooth maps, performing as well as if we knew the smoothness level $\alpha$ ahead of time. Our result sheds light on architecture design for deep neural networks, namely the choice of depth, width and sparsity level. These network attributes typically depend on unknown smoothness in order to be optimal. We obviate this constraint with the fully Bayes construction. As an aside, we show that SS-DL does not overfit in the sense that the posterior concentrates on smaller networks with fewer (up to the optimal number of) nodes and links. Our results provide new theoretical justifications for deep ReLU networks from a Bayesian point of view.

该论文研究了一种名为 Spike-and-Slab Deep Learning 的全新深度学习建模方法，该方法采用完全 Bayes 的方式代替 Dropout 正则化，使得深度神经网络能够拥有更好的泛化性能，并且无需事先了解平滑度即可实现顺畅的输入输出映射。此外，该方法提供了深度 ReLU 神经网络的全新理论解释，有望改变未来神经网络的设计与构建方式。

稀疏深度学习的后验集中