We study derivative-free methods for policy optimization over the class of linear policies. We focus on characterizing the convergence rate of these methods when applied to linear-quadratic systems, and study various settings of driving noise and reward feedback. We show that these methods provably converge to within any pre-specified tolerance of the optimal policy with a number of zero-order evaluations that is an explicit polynomial of the error tolerance, dimension, and curvature properties of the problem. Our analysis reveals some interesting differences between the settings of additive driving noise and random initialization, as well as the settings of one-point and two-point reward feedback. Our theory is corroborated by extensive simulations of derivative-free methods on these systems. Along the way, we derive convergence rates for stochastic zero-order optimization algorithms when applied to a certain class of non-convex problems.

本文研究了在线性策略的类别中，基于无导数法的策略优化方法。研究了不同的驱动噪声和奖励反馈设置，特别是应用于线性二次系统时的收敛速度，发现这些方法会在求解问题的误差、维度和曲率特性的确定的多项式次零阶求解下收敛至最优解水平，并发现了不同驱动噪声和奖励反馈设置下的一些有趣差异，最终在对这些系统进行广泛的模拟验证下进行计算。此外，我们也研究了基于零阶优化算法的随机非凸问题的收敛速度。

无导数策略优化方法：线性二次系统的保证