While stochastic gradient descent (SGD) is still the most popular
optimization algorithm in deep learning, adaptive algorithms such as Adam have
established empirical advantages over SGD in some deep learning applications
such as training transformers. However, it remains a question that why Adam
converges significantly faster than SGD in these scenarios. In this paper, we
propose one explanation of why Adam converges faster than SGD using a new
concept directional sharpness. We argue that the performance of optimization
algorithms is closely related to the directional sharpness of the update steps,
and show SGD has much worse directional sharpness compared to adaptive
algorithms. We further observe that only a small fraction of the coordinates
causes the bad sharpness and slow convergence of SGD, and propose to use
coordinate-wise clipping as a solution to SGD and other optimization
algorithms. We demonstrate the effect of coordinate-wise clipping on sharpness
reduction and speeding up the convergence of optimization algorithms under
various settings. We show that coordinate-wise clipping improves the local loss
reduction when only a small fraction of the coordinates has bad sharpness. We
conclude that the sharpness reduction effect of adaptive coordinate-wise
scaling is the reason for Adam's success in practice and suggest the use of
coordinate-wise clipping as a universal technique to speed up deep learning
optimization.

这篇论文提出了一种新的方向锐度概念，阐述了优化算法与更新步长的方向锐度的关系，发现随机梯度下降在这方面表现远差于自适应算法，因此建议使用坐标剪裁该问题，并证明了该技术可用于提高深度学习优化的收敛速度。