BriefGPT.xyz
May, 2023
有偏差SGD指南
A Guide Through the Zoo of Biased SGD
HTML
PDF
Yury Demidovich, Grigory Malinovsky, Igor Sokolov, Peter Richtárik
TL;DR
本文分析了带偏估计器的随机梯度下降(BiasedSGD)算法在凸和非凸环境下的效果并比较了带偏估计器和无偏估计器的优缺点,同时提出了一组新的比以往任何假设更弱的假设,并通过实验结果验证了理论发现。
Abstract
stochastic gradient descent
(SGD) is arguably the most important single algorithm in modern machine learning. Although SGD with unbiased gradient estimators has been studied extensively over at least half a century, SGD variants relying on
→