As application demands for zeroth-order (gradient-free) optimization accelerate, the need for variance reduced and faster converging approaches is also intensifying. This paper addresses these challenges by presenting: a) a comprehensive theoretical analysis of variance reduced zeroth-order (ZO) optimization, b) a novel variance reduced ZO algorithm, called ZO-SVRG, and c) an experimental evaluation of our approach in the context of two compelling applications, black-box chemical material classification and generation of adversarial examples from black-box deep neural network models. Our theoretical analysis uncovers an essential difficulty in the analysis of ZO-SVRG: the unbiased assumption on gradient estimates no longer holds. We prove that compared to its first-order counterpart, ZO-SVRG with a two-point random gradient estimator suffers an additional error of order $O(1/b)$, where $b$ is the mini-batch size. To mitigate this error, we propose two accelerated versions of ZO-SVRG utilizing variance reduced gradient estimators, which achieve the best rate known for ZO stochastic optimization (in terms of iterations). Our extensive experimental results show that our approaches outperform other state-of-the-art ZO algorithms, and strike a balance between the convergence rate and the function query complexity.

本文提出了一种基于损失函数的零阶优化算法（ZO-SVRG）以及相应的快速收敛的优化方法，即方差减少，用于解决应用中需要零阶优化的挑战，可用于黑盒物质分类和黑盒深度神经网络模型生成对抗性示例。我们的理论分析揭示了ZO-SVRG的本质难点，并提出了两种利用方差减少的梯度估计器的加速版本ZO-SVRG，其在ZO随机优化（迭代次数）方面具有已知的最佳速度。与当前其他最先进的ZO算法相比，我们的方法在功能查询复杂性和收敛速率之间取得了平衡。

零阶随机方差约减用于非凸优化