We study the implicit bias of the general family of steepest descent algorithms, which includes gradient descent, sign descent and coordinate descent, in deep homogeneous neural networks. We prove that an algorithm-dependent geometric margin starts increasing once the networks reach perfect training accuracy and characterize the late-stage bias of the algorithms. In particular, we define a generalized notion of stationarity for optimization problems and show that the algorithms progressively reduce a (generalized) Bregman divergence, which quantifies proximity to such stationary points of a margin-maximization problem. We then experimentally zoom into the trajectories of neural networks optimized with various steepest descent algorithms, highlighting connections to the implicit bias of Adam.

本研究探讨了包括梯度下降、符号下降和坐标下降在内的最陡下降法的隐性偏差，填补了在深度均匀神经网络中这一领域的研究空白。我们提出了一种优化问题的广义平稳性概念，并证明这些算法在达到完美训练准确度后，几何边际会开始增加，进而逐步减少广义的Bregman散度，为理解深度学习中的算法偏差提供了新的视角和实证支持。

边际的特性：均匀神经网络中最陡下降法的隐性偏差