Policy gradient methods have enjoyed great success in deep reinforcement
learning but suffer from high variance of gradient estimates. The high variance
problem is particularly exasperated in problems with long horizons or
high-dimensional action spaces. To mitigate this issue, we derive a bias-free
action-dependent baseline for variance reduction which fully exploits the
structural form of the stochastic policy itself and does not make any
additional assumptions about the MDP. We demonstrate and quantify the benefit
of the action-dependent baseline through both theoretical analysis as well as
numerical results, including an analysis of the suboptimality of the optimal
state-dependent baseline. The result is a computationally efficient policy
gradient algorithm, which scales to high-dimensional control problems, as
demonstrated by a synthetic 2000-dimensional target matching task. Our
experimental results indicate that action-dependent baselines allow for faster
learning on standard reinforcement learning benchmarks and high-dimensional
hand manipulation and synthetic tasks. Finally, we show that the general idea
of including additional information in baselines for improved variance
reduction can be extended to partially observed and multi-agent tasks.

本研究提出了一种无偏差的基于动作的基线方法，该方法可以减少深度增强学习中梯度估计的高方差问题，最终实现高维控制问题的策略梯度算法。此外，还证明了该方法的有效性并扩展到部分观察和多智能体任务。