In distributed deep learning, a large batch size in Stochastic Gradient Descent is required to fully exploit the computing power in distributed systems. However, generalization gap (accuracy loss) was observed because large-batch training converges to sharp minima which have bad generalization [1][2]. This contradiction hinders the scalability of distributed deep learning. We propose SmoothOut to smooth out sharp minima in Deep Neural Networks (DNNs) and thereby close generalization gap. SmoothOut perturbs multiple copies of the DNN in the parameter space and averages these copies. We prove that SmoothOut can eliminate sharp minima. Perturbing and training multiple DNN copies is inefficient, we propose a stochastic version of SmoothOut which only introduces overhead of noise injection and denoising per iteration. We prove that the Stochastic SmoothOut is an unbiased approximation of the original SmoothOut. In experiments on a variety of DNNs and datasets, SmoothOut consistently closes generalization gap in large-batch training within the same epochs. Moreover, SmoothOut can guide small-batch training to flatter minima and improve generalization. Our source code is in https://github.com/wenwei202/smoothout

本研究提出了SmoothOut框架，通过注入噪声来平滑深度神经网络中的锐利极小值，从而提高了广义泛化性能。在实验中，SmoothOut和AdaSmoothOut在小批量和大批量训练中均稳定提高了广义化能力。

SmoothOut: 消除深度学习中的尖峰最小值以提高泛化能力