TL;DR本文提出了一种快速的随机拟牛顿方法,针对平滑性不均匀的情况,通过梯度剪切和方差减小,实现了最优的 O (ε^(-3)) 样本复杂度,并通过简单的超参数调节实现了收敛加速,数值实验证明了该算法优于现有方法。
Abstract
Classical convergence analyses for optimization algorithms rely on the
widely-adopted uniform smoothness assumption. However, recent experimental
studies have demonstrated that many machine learning problems exhibit
non-uniform smoothness, meaning the smoothness factor is a function of