Model-based reinforcement learning (MBRL) has gained much attention for its
ability to learn complex behaviors in a sample-efficient way: planning actions
by generating imaginary trajectories with predicted rewards. Despite its
success, we found that surprisingly, reward prediction is often a bottleneck of
MBRL, especially for sparse rewards that are challenging (or even ambiguous) to
predict. Motivated by the intuition that humans can learn from rough reward
estimates, we propose a simple yet effective reward smoothing approach,
DreamSmooth, which learns to predict a temporally-smoothed reward, instead of
the exact reward at the given timestep. We empirically show that DreamSmooth
achieves state-of-the-art performance on long-horizon sparse-reward tasks both
in sample efficiency and final performance without losing performance on common
benchmarks, such as Deepmind Control Suite and Atari benchmarks.

基于模型的强化学习（MBRL）通过生成带有预测奖励的虚拟轨迹来规划动作，以高样本效率地学习复杂行为。我们提出一种简单而有效的奖励平滑方法 DreamSmooth，通过学习预测时间上平滑的奖励，而不是给定时间步的精确奖励。实证结果表明，DreamSmooth 在长时间范围的稀疏奖励任务上实现了最先进的性能，同时在常见的基准测试中也不损失性能，如 Deepmind 控制套件和 Atari 基准。