Quasi-Newton methods still face significant challenges in training
large-scale neural networks due to additional compute costs in the Hessian
related computations and instability issues in stochastic training. A
well-known method, L-BFGS that efficiently approximates the Hessian using
history parameter and gradient changes, suffers convergence instability in
stochastic training. So far, attempts that adapt L-BFGS to large-scale
stochastic training incur considerable extra overhead, which offsets its
convergence benefits in wall-clock time. In this paper, we propose mL-BFGS, a
lightweight momentum-based L-BFGS algorithm that paves the way for quasi-Newton
(QN) methods in large-scale distributed deep neural network (DNN) optimization.
mL-BFGS introduces a nearly cost-free momentum scheme into L-BFGS update and
greatly reduces stochastic noise in the Hessian, therefore stabilizing
convergence during stochastic optimization. For model training at a large
scale, mL-BFGS approximates a block-wise Hessian, thus enabling distributing
compute and memory costs across all computing nodes. We provide a supporting
convergence analysis for mL-BFGS in stochastic settings. To investigate mL-BFGS
potential in large-scale DNN training, we train benchmark neural models using
mL-BFGS and compare performance with baselines (SGD, Adam, and other
quasi-Newton methods). Results show that mL-BFGS achieves both noticeable
iteration-wise and wall-clock speedup.

mL-BFGS 是一种基于动量的轻量级 L-BFGS 算法，为大规模分布式深度神经网络 (DNN) 优化带来了拟牛顿法在大规模训练中的潜力。mL-BFGS 通过引入几乎无额外计算成本的动量方案，并显著减少 Hessian 中的随机噪声，从而在随机优化过程中稳定收敛。实验结果表明，mL-BFGS 在迭代和实际时间上均获得了明显的加速效果，并与基准方法（SGD、Adam 和其他拟牛顿法）进行了比较。