The big breakthrough on the ImageNet challenge in 2012 was partially due to the `dropout' technique used to avoid overfitting. Here, we introduce a new approach called `Spectral Dropout' to improve the generalization ability of deep neural networks. We cast the proposed approach in the form of regular Convolutional Neural Network (CNN) weight layers using a decorrelation transform with fixed basis functions. Our spectral dropout method prevents overfitting by eliminating weak and `noisy' Fourier domain coefficients of the neural network activations, leading to remarkably better results than the current regularization methods. Furthermore, the proposed is very efficient due to the fixed basis functions used for spectral transformation. In particular, compared to Dropout and Drop-Connect, our method significantly speeds up the network convergence rate during the training process (roughly x2), with considerably higher neuron pruning rates (an increase of ~ 30%). We demonstrate that the spectral dropout can also be used in conjunction with other regularization approaches resulting in additional performance gains.

提出了一种称为`Spectral Dropout'的新方法，它可以通过固定基函数的正交变换的形式将其转化为正则卷积神经网络(CNN)的权重层，通过消除神经网络激活的弱且嘈杂的傅里叶系数来预防过拟合，相比当前正则化方法具有显著提高网络收敛速度以及神经元精简率的效果，并且可以与其他正则化方法共同使用。

谱 Dropout 正则化深度神经网络