BriefGPT.xyz
Nov, 2020
方向很重要:关于中等学习率的随机梯度下降的隐式偏差
Direction Matters: On the Implicit Regularization Effect of Stochastic Gradient Descent with Moderate Learning Rate
HTML
PDF
Jingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu
TL;DR
本研究针对模型学习速率为中等并逐渐降低的情况,研究了SGD和GD在超参数调节中的常见行为,以此试图解决机器学习中的算法偏差问题,并得出了不同方向偏差可能导致最终预测结果差异的结论。
Abstract
Understanding the algorithmic
regularization
effect of
stochastic gradient descent
(SGD) is one of the key challenges in modern machine learning and deep learning theory. Most of the existing works, however, focu
→