In this paper, we study a class of stochastic bilevel optimization problems,
also known as stochastic simple bilevel optimization, where we minimize a
smooth stochastic objective function over the optimal solution set of another
stochastic convex optimization problem. We introduce novel stochastic bilevel
optimization methods that locally approximate the solution set of the
lower-level problem via a stochastic cutting plane, and then run a conditional
gradient update with variance reduction techniques to control the error induced
by using stochastic gradients. For the case that the upper-level function is
convex, our method requires
$\tilde{\mathcal{O}}(\max\{1/\epsilon_f^{2},1/\epsilon_g^{2}\}) $ stochastic
oracle queries to obtain a solution that is $\epsilon_f$-optimal for the
upper-level and $\epsilon_g$-optimal for the lower-level. This guarantee
improves the previous best-known complexity of
$\mathcal{O}(\max\{1/\epsilon_f^{4},1/\epsilon_g^{4}\})$. Moreover, for the
case that the upper-level function is non-convex, our method requires at most
$\tilde{\mathcal{O}}(\max\{1/\epsilon_f^{3},1/\epsilon_g^{3}\}) $ stochastic
oracle queries to find an $(\epsilon_f, \epsilon_g)$-stationary point. In the
finite-sum setting, we show that the number of stochastic oracle calls required
by our method are $\tilde{\mathcal{O}}(\sqrt{n}/\epsilon)$ and
$\tilde{\mathcal{O}}(\sqrt{n}/\epsilon^{2})$ for the convex and non-convex
settings, respectively, where $\epsilon=\min \{\epsilon_f,\epsilon_g\}$.

我们研究了一类随机二级优化问题，引入了一种新颖的随机二级优化方法，通过随机切割平面局部逼近下层问题的解集，并运行带有方差减少技术的条件梯度更新来控制由于使用随机梯度引起的误差。在上层函数为凸函数的情况下，我们的方法需要最多 $\tilde {\mathcal {O}}(\max\{1/\epsilon_f^{2},1/\epsilon_g^{2}\})$ 个随机 oracle 查询，得到一个上层函数精度为 $\epsilon_f$、下层函数精度为 $\epsilon_g$ 的解，这一保证改进了之前已知的复杂性 $\mathcal {O}(\max\{1/\epsilon_f^{4},1/\epsilon_g^{4}\})$。此外，在上层函数为非凸函数的情况下，我们的方法需要最多 $\tilde {\mathcal {O}}(\max\{1/\epsilon_f^{3},1/\epsilon_g^{3}\})$ 个随机 oracle 查询，找到一个 $(\epsilon_f, \epsilon_g)$- 稳定点。在有限和问题设置中，对于凸和非凸情况下，我们的方法所需的随机 oracle 调用次数分别为 $\tilde {\mathcal {O}}(\sqrt {n}/\epsilon)$ 和 $\tilde {\mathcal {O}}(\sqrt {n}/\epsilon^{2})$，其中 $\epsilon=\min \{\epsilon_f,\epsilon_g\}$。