We propose expected policy gradients (EPG), which unify stochastic policy
gradients (SPG) and deterministic policy gradients (DPG) for reinforcement
learning. Inspired by expected sarsa, EPG integrates (or sums) across actions
when estimating the gradient, instead of relying only on the action in the
sampled trajectory. For continuous action spaces, we first derive a practical
result for Gaussian policies and quadratic critics and then extend it to a
universal analytical method, covering a broad class of actors and critics,
including Gaussian, exponential families, and policies with bounded support.
For Gaussian policies, we introduce an exploration method that uses covariance
proportional to the matrix exponential of the scaled Hessian of the critic with
respect to the actions. For discrete action spaces, we derive a variant of EPG
based on softmax policies. We also establish a new general policy gradient
theorem, of which the stochastic and deterministic policy gradient theorems are
special cases. Furthermore, we prove that EPG reduces the variance of the
gradient estimates without requiring deterministic policies and with little
computational overhead. Finally, we provide an extensive experimental
evaluation of EPG and show that it outperforms existing approaches on multiple
challenging control domains.

提出了期望策略梯度（EPG）方法，将随机策略梯度（SPG）和确定性策略梯度（DPG）方法统一起来，用于连续或离散动作空间的强化学习中，实验证明其在多项控制任务中胜过现有方法。