Policy-gradient algorithms are effective reinforcement learning methods for solving control problems with continuous state and action spaces. To compute near-optimal policies, it is essential in practice to include exploration terms in the learning objective. Although the effectiveness of these terms is usually justified by an intrinsic need to explore environments, we propose a novel analysis and distinguish two different implications of these techniques. First, they make it possible to smooth the learning objective and to eliminate local optima while preserving the global maximum. Second, they modify the gradient estimates, increasing the probability that the stochastic parameter update eventually provides an optimal policy. In light of these effects, we discuss and illustrate empirically exploration strategies based on entropy bonuses, highlighting their limitations and opening avenues for future works in the design and analysis of such strategies.

我们提出了一种新的分析方法，并区分了探索技术的两个不同影响。首先，它们使得能够平滑学习目标并消除局部最优解，同时保留全局最优解。其次，它们修改了梯度估计，增加了随机参数更新最终提供最优策略的概率。在这些影响的基础上，我们讨论并通过熵奖励进行了实证研究，突出了其局限性，并为该策略的设计和分析开辟了未来的研究方向。

政策梯度背后的探索神话