This paper explores adaptive variance reduction methods for stochastic
optimization based on the STORM technique. Existing adaptive extensions of
STORM rely on strong assumptions like bounded gradients and bounded function
values, or suffer an additional $\mathcal{O}(\log T)$ term in the convergence
rate. To address these limitations, we introduce a novel adaptive STORM method
that achieves an optimal convergence rate of $\mathcal{O}(T^{-1/3})$ for
non-convex functions with our newly designed learning rate strategy. Compared
with existing approaches, our method requires weaker assumptions and attains
the optimal convergence rate without the additional $\mathcal{O}(\log T)$ term.
We also extend the proposed technique to stochastic compositional optimization,
obtaining the same optimal rate of $\mathcal{O}(T^{-1/3})$. Furthermore, we
investigate the non-convex finite-sum problem and develop another innovative
adaptive variance reduction method that achieves an optimal convergence rate of
$\mathcal{O}(n^{1/4} T^{-1/2} )$, where $n$ represents the number of component
functions. Numerical experiments across various tasks validate the
effectiveness of our method.

本篇论文探讨了基于 STORM 技术的自适应方差缩减方法。通过引入新设计的学习率策略，我们提出了一种新颖的自适应 STORM 方法，该方法对于非凸函数达到了最佳收敛率 O (T^-1/3)，同时无需额外的 O (log T) 项。我们还将该技术扩展到随机组合优化中，获得了相同的最佳收敛率 O (T^-1/3)。此外，我们还研究了非凸有限和问题，并开发了另一种创新的自适应方差缩减方法，该方法实现了最佳收敛率 O (n^1/4T^-1/2)，其中 n 表示组成函数的数量。通过各种任务的数值实验验证了我们方法的有效性。