We present a new method to reduce the variance of stochastic versions of the BFGS optimization method, applied to the optimization of a class of smooth strongly convex functions. Although Stochastic Gradient Descent (SGD) is a popular method to solve this kind of problem, its convergence rate is sublinear as it is in fact limited by the noisy approximation of the true gradient. In order to recover a high convergence rate, one has to pick an appropriate step-size or explicitly reduce the variance of the approximate gradients. Another limiting factor of SGD is that it ignores the curvature of the objective function that can help greatly speed up convergence. Stochastic variants of BFGS that include curvature have shown good empirical performance but suffer from the same noise effects as SGD. We here propose a new algorithm V ITE that uses an existing technique to reduce this variance while allowing a constant step-size to be used. We show that the expected objective value converges to the optimum at a geometric rate. We experimentally demonstrate improved convergence rate on diverse stochastic optimization problems.

该研究提出了一种名为“Vite”的基于Stochastic Quasi-Newton算法的优化方法，它利用一种现有的一阶技术来减少噪声和方差，并在大规模学习问题上取得了不错的结果。

一种方差减少的随机牛顿法