Several low-bandwidth distributable black-box optimization algorithms in the
family of finite differences such as Evolution Strategies have recently been
shown to perform nearly as well as tailored Reinforcement Learning methods in
some Reinforcement Learning domains. One shortcoming of these black-box methods
is that they must collect information about the structure of the return
function at every update, and can often employ only information drawn from a
distribution centered around the current parameters. As a result, when these
algorithms are distributed across many machines, a significant portion of total
runtime may be spent with many machines idle, waiting for a final return and
then for an update to be calculated. In this work we introduce a novel method
to use older data in finite difference algorithms, which produces a scalable
algorithm that avoids significant idle time or wasted computation.

本研究提出了一种将旧数据运用于有限差分算法中的新方法，从而产生出一种可伸缩且避免空闲时间或浪费计算的算法，解决了分布式黑盒优化算法的瓶颈问题。

深度强化学习的可扩展有限差分方法

A Scalable Finite Difference Method for Deep Reinforcement Learning

In this paper, we analyze several methods for approximating gradients of
noisy functions using only function values. These methods include finite
differences, linear interpolation, Gaussian smoothing and smoothing on a
sphere. The methods differ in the number of functions sampled, the choice of
the sample points, and the way in which the gradient approximations are
derived. For each method, we derive bounds on the number of samples and the
sampling radius which guarantee favorable convergence properties for a line
search or fixed step size descent method. To this end, we use the results in
[Berahas et al., 2019] and show how each method can satisfy the sufficient
conditions, possibly only with some sufficiently large probability at each
iteration, as happens to be the case with Gaussian smoothing and smoothing on a
sphere. Finally, we present numerical results evaluating the quality of the
gradient approximations as well as their performance in conjunction with a line
search derivative-free optimization algorithm.

本文分析了多个仅基于函数值的逼近噪声函数梯度的方法，包括有限差分、线性插值、高斯平滑和球面平滑等，并给出了收敛性保证和数值结果以及应用于无导数优化算法的表现。