Long-term temporal credit assignment is an important challenge in deep
reinforcement learning (RL). It refers to the ability of the agent to attribute
actions to consequences that may occur after a long time interval. Existing
policy-gradient and Q-learning algorithms typically rely on dense environmental
rewards that provide rich short-term supervision and help with credit
assignment. However, they struggle to solve tasks with delays between an action
and the corresponding rewarding feedback. To make credit assignment easier,
recent works have proposed algorithms to learn dense "guidance" rewards that
could be used in place of the sparse or delayed environmental rewards. This
paper is in the same vein -- starting with a surrogate RL objective that
involves smoothing in the trajectory-space, we arrive at a new algorithm for
learning guidance rewards. We show that the guidance rewards have an intuitive
interpretation, and can be obtained without training any additional neural
networks. Due to the ease of integration, we use the guidance rewards in a few
popular algorithms (Q-learning, Actor-Critic, Distributional-RL) and present
results in single-agent and multi-agent tasks that elucidate the benefit of our
approach when the environmental rewards are sparse or delayed.

该论文介绍了一种使用轨迹空间平滑来学习指导奖励的算法，并阐明了该算法在解决强化学习中长期时序信用分配问题上的优越性。