Recent research has observed that in machine learning optimization, gradient
descent (GD) often operates at the edge of stability (EoS) [Cohen, et al.,
2021], where the stepsizes are set to be large, resulting in non-monotonic
losses induced by the GD iterates. This paper studies the convergence and
implicit bias of constant-stepsize GD for logistic regression on linearly
separable data in the EoS regime. Despite the presence of local oscillations,
we prove that the logistic loss can be minimized by GD with any constant
stepsize over a long time scale. Furthermore, we prove that with any constant
stepsize, the GD iterates tend to infinity when projected to a max-margin
direction (the hard-margin SVM direction) and converge to a fixed vector that
minimizes a strongly convex potential when projected to the orthogonal
complement of the max-margin direction. In contrast, we also show that in the
EoS regime, GD iterates may diverge catastrophically under the exponential
loss, highlighting the superiority of the logistic loss. These theoretical
findings are in line with numerical simulations and complement existing
theories on the convergence and implicit bias of GD, which are only applicable
when the stepsizes are sufficiently small.

本文研究了边缘稳定性（EoS）中逻辑回归上梯度下降（GD）的收敛和隐式偏差情况，证明任何恒定步长的非单调 GD 迭代可以在较长时间尺度上最小化逻辑损失，并在最大间隔方向上趋于正无穷，在最大间隔方向的正交补上收敛于最小化强凸势能的固定向量，而指数损失可能导致 GD 迭代在 EoS 区域内灾难性发散。