We propose a remarkably general variance-reduced method suitable for solving regularized empirical risk minimization problems with either a large number of training examples, or a large model dimension, or both. In special cases, our method reduces to several known and previously thought to be unrelated methods, such as {\tt SAGA}, {\tt LSVRG}, {\tt JacSketch}, {\tt SEGA} and {\tt ISEGA}, and their arbitrary sampling and proximal generalizations. However, we also highlight a large number of new specific algorithms with interesting properties. We provide a single theorem establishing linear convergence of the method under smoothness and quasi strong convexity assumptions. With this theorem we recover best-known and sometimes improved rates for known methods arising in special cases. As a by-product, we provide the first unified method and theory for stochastic gradient and stochastic coordinate descent type methods.

提出了一种通用的降方差的方法，适用于解决带有大量训练样例或大型模型维度或两者都有的正则化经验风险最小化问题。该方法可以减少已知的多种方法，同时提供了一种单一的定理，该定理可以证明在平滑和拟强凸性假设下的线性收敛性。此外，该方法还为随机梯度和随机坐标下降等方法提供了首个统一的方法和理论。

一种支配所有的方法：用于数据、参数和多种新方法的方差缩减