Reward shaping is an effective technique for incorporating domain knowledge into reinforcement learning (RL). Existing approaches such as potential-based reward shaping normally make full use of a given shaping reward function. However, since the transformation of human knowledge into numeric reward values is often imperfect due to reasons such as human cognitive bias, completely utilizing the shaping reward function may fail to improve the performance of RL algorithms. In this paper, we consider the problem of adaptively utilizing a given shaping reward function. We formulate the utilization of shaping rewards as a bi-level optimization problem, where the lower level is to optimize policy using the shaping rewards and the upper level is to optimize a parameterized shaping weight function for true reward maximization. We formally derive the gradient of the expected true reward with respect to the shaping weight function parameters and accordingly propose three learning algorithms based on different assumptions. Experiments in sparse-reward cartpole and MuJoCo environments show that our algorithms can fully exploit beneficial shaping rewards, and meanwhile ignore unbeneficial shaping rewards or even transform them into beneficial ones.

本文提出了一种自适应利用给定塑形奖励函数的算法，通过将塑形奖励作为一个双层优化问题来解决，从而实现了真实奖励的最大化，并基于这个问题，提出了三种基于不同假设的学习算法。实验结果表明，我们的算法可以充分利用有益的塑形奖励，同时忽略无益的塑形奖励或者甚至将它们转化为有益的。

学习如何利用成形奖励：一种新的奖励成形方法