Recently there has been a proliferation of intrinsic motivation (IM)
reward-shaping methods to learn in complex and sparse-reward environments.
These methods can often inadvertently change the set of optimal policies in an
environment, leading to suboptimal behavior. Previous work on mitigating the
risks of reward shaping, particularly through potential-based reward shaping
(PBRS), has not been applicable to many IM methods, as they are often complex,
trainable functions themselves, and therefore dependent on a wider set of
variables than the traditional reward functions that PBRS was developed for. We
present an extension to PBRS that we prove preserves the set of optimal
policies under a more general set of functions than has been previously proven.
We also present {\em Potential-Based Intrinsic Motivation} (PBIM), a method for
converting IM rewards into a potential-based form that is useable without
altering the set of optimal policies. Testing in the MiniGrid DoorKey and Cliff
Walking environments, we demonstrate that PBIM successfully prevents the agent
from converging to a suboptimal policy and can speed up training.

最近出现了大量的内在动机（IM）奖励塑造方法来学习复杂和稀疏奖励的环境。在这些方法中，往往会无意中改变环境中的最优策略集，导致次优行为。我们提出了一个扩展的潜在基于奖励塑造（PBRS）方法，我们证明它在比之前更一般的函数集合下保留了最优策略集。同时，我们还提出了一种名为 “基于潜在的内在动机”（PBIM）的方法，将 IM 奖励转化为基于潜在的形式，而无需改变最优策略集。在 MiniGrid DoorKey 和 Cliff Walking 环境中的测试中，我们证明 PBIM 成功防止了智能体收敛到次优策略并且可以加速训练。