The recent advanced evolution-based zeroth-order optimization methods and the
policy gradient-based first-order methods are two promising alternatives to
solve reinforcement learning (RL) problems with complementary advantages. The
former methods work with arbitrary policies, drive state-dependent and
temporally-extended exploration, possess robustness-seeking property, but
suffer from high sample complexity, while the latter methods are more sample
efficient but are restricted to differentiable policies and the learned
policies are less robust. To address these issues, we propose a novel
Zeroth-Order Actor-Critic algorithm (ZOAC), which unifies these two methods
into an on-policy actor-critic architecture to preserve the advantages from
both. ZOAC conducts rollouts collection with timestep-wise perturbation in
parameter space, first-order policy evaluation (PEV) and zeroth-order policy
improvement (PIM) alternately in each iteration. We extensively evaluate our
proposed method on a wide range of challenging continuous control benchmarks
using different types of policies, where ZOAC outperforms zeroth-order and
first-order baseline algorithms.

提出了一种新的零阶演员评论家算法（ZOAC），将进化型的零阶优化方法和基于政策梯度的一阶方法统一到一个政策演员结构中，解决了两种算法的缺陷，获得了更高的样本效率和更强的鲁棒性。