Classical convergence analyses for optimization algorithms rely on the
widely-adopted uniform smoothness assumption. However, recent experimental
studies have demonstrated that many machine learning problems exhibit
non-uniform smoothness, meaning the smoothness factor is a function of the
model parameter instead of a universal constant. In particular, it has been
observed that the smoothness grows with respect to the gradient norm along the
training trajectory. Motivated by this phenomenon, the recently introduced
$(L_0, L_1)$-smoothness is a more general notion, compared to traditional
$L$-smoothness, that captures such positive relationship between smoothness and
gradient norm. Under this type of non-uniform smoothness, existing literature
has designed stochastic first-order algorithms by utilizing gradient clipping
techniques to obtain the optimal $\mathcal{O}(\epsilon^{-3})$ sample complexity
for finding an $\epsilon$-approximate first-order stationary solution.
Nevertheless, the studies of quasi-Newton methods are still lacking.
Considering higher accuracy and more robustness for quasi-Newton methods, in
this paper we propose a fast stochastic quasi-Newton method when there exists
non-uniformity in smoothness. Leveraging gradient clipping and variance
reduction, our algorithm can achieve the best-known
$\mathcal{O}(\epsilon^{-3})$ sample complexity and enjoys convergence speedup
with simple hyperparameter tuning. Our numerical experiments show that our
proposed algorithm outperforms the state-of-the-art approaches.

本文提出了一种快速的随机拟牛顿方法，针对平滑性不均匀的情况，通过梯度剪切和方差减小，实现了最优的 O (ε^(-3)) 样本复杂度，并通过简单的超参数调节实现了收敛加速，数值实验证明了该算法优于现有方法。

一种非凸优化的随机拟牛顿方法

A Stochastic Quasi-Newton Method for Non-convex Optimization with  Non-uniform Smoothness

The question of how to incorporate curvature information in stochastic
approximation methods is challenging. The direct application of classical
quasi- Newton updating techniques for deterministic optimization leads to noisy
curvature estimates that have harmful effects on the robustness of the
iteration. In this paper, we propose a stochastic quasi-Newton method that is
efficient, robust and scalable. It employs the classical BFGS update formula in
its limited memory form, and is based on the observation that it is beneficial
to collect curvature information pointwise, and at regular intervals, through
(sub-sampled) Hessian-vector products. This technique differs from the
classical approach that would compute differences of gradients, and where
controlling the quality of the curvature estimates can be difficult. We present
numerical results on problems arising in machine learning that suggest that the
proposed method shows much promise.

本文提出了一种基于限制记忆的 BFGS 更新公式和子采样 Hessian - 向量积的随机拟牛顿方法来有效地、稳健地和可伸缩地处理如何将曲率信息纳入随机逼近方法的问题，并通过机器学习问题上的数值结果展示其前景。