Direct policy optimization in reinforcement learning is usually solved with
policy-gradient algorithms, which optimize policy parameters via stochastic
gradient ascent. This paper provides a new theoretical interpretation and
justification of these algorithms. First, we formulate direct policy
optimization in the optimization by continuation framework. The latter is a
framework for optimizing nonconvex functions where a sequence of surrogate
objective functions, called continuations, are locally optimized. Second, we
show that optimizing affine Gaussian policies and performing entropy
regularization can be interpreted as implicitly optimizing deterministic
policies by continuation. Based on these theoretical results, we argue that
exploration in policy-gradient algorithms consists in computing a continuation
of the return of the policy at hand, and that the variance of policies should
be history-dependent functions adapted to avoid local extrema rather than to
maximize the return of the policy.

本研究提供了关于策略梯度算法的新的理论解释和证明。同时，文章还提出，策略梯度算法中的探索是计算当前政策回报的连续，而策略的方差应该是适应历史的函数，以避免局部极值，而不是为了最大化政策的回报。