Finding parameters that minimise a loss function is at the core of many machine learning methods. The Stochastic Gradient Descent algorithm is widely used and delivers state of the art results for many problems. Nonetheless, Stochastic Gradient Descent typically cannot find the global minimum, thus its empirical effectiveness is hitherto mysterious. We derive a correspondence between parameter inference and free energy minimisation in statistical physics. The degree of undersampling plays the role of temperature. Analogous to the energy-entropy competition in statistical physics, wide but shallow minima can be optimal if the system is undersampled, as is typical in many applications. Moreover, we show that the stochasticity in the algorithm has a non-trivial correlation structure which systematically biases it towards wide minima. We illustrate our argument with two prototypical models: image classification using deep learning, and a linear neural network where we can analytically reveal the relationship between entropy and out-of-sample error.

研究发现，针对机器学习中许多问题，大多数采用随机梯度下降算法，并能够在实践中实现最佳结果，但通常无法达到全局最小值，其实际效果至今仍是一个谜，本文研究了参数推断和统计物理中的自由能最小化之间的对应关系，证明了宽而浅的极小值在系统欠采样时是最优的，同时还显示该算法的随机性具有非平凡的相关结构，会使其有偏地偏向于宽的极小值。

能量熵竞争及其对随机梯度下降在机器学习中的有效性影响