Recent success in Deep Reinforcement Learning (DRL) methods has shown that
policy optimization with respect to an off-policy distribution via importance
sampling is effective for sample reuse. In this paper, we show that the use of
importance sampling could introduce high variance in the objective estimate.
Specifically, we show in a principled way that the variance of importance
sampling estimate grows quadratically with importance ratios and the large
ratios could consequently jeopardize the effectiveness of surrogate objective
optimization. We then propose a technique called sample dropout to bound the
estimation variance by dropping out samples when their ratio deviation is too
high. We instantiate this sample dropout technique on representative policy
optimization algorithms, including TRPO, PPO, and ESPO, and demonstrate that it
consistently boosts the performance of those DRL algorithms on both continuous
and discrete action controls, including MuJoCo, DMControl and Atari video
games. Our code is open-sourced at
https://github.com/LinZichuan/sdpo.git.

本文通过实验表明，利用重要性抽样进行策略优化的代价是很高的方差估计，提出了一种名为 “样本丢弃” 的技术，以限制重要性抽样带来的估计方差，并在一些代表性的策略优化算法上应用，如 TRPO、PPO 和 ESPO，实验证明 “样本丢弃” 可以提高这些深度强化学习算法的性能。