How much credit (or blame) should an action taken in a state get for a future reward? This is the fundamental temporal credit assignment problem in Reinforcement Learning (RL). One of the earliest and still most widely used heuristics is to assign this credit based on a scalar coefficient $\lambda$ (treated as a hyperparameter) raised to the power of the time interval between the state-action and the reward. In this empirical paper, we explore heuristics based on more general pairwise weightings that are functions of the state in which the action was taken, the state at the time of the reward, as well as the time interval between the two. Of course it isn't clear what these pairwise weight functions should be, and because they are too complex to be treated as hyperparameters we develop a metagradient procedure for learning these weight functions during the usual RL training of a policy. Our empirical work shows that it is often possible to learn these pairwise weight functions during learning of the policy to achieve better performance than competing approaches.

本文旨在解决强化学习中最基本的关于时间信用分配问题，通过使用基于状态时的换算系数或基于更一般的由状态，所需格外步骤和奖励时间之间的函数关系的静态/动态配重方法，在学习RL策略的过程中使用元梯度方法学习这些分配函数从而提高性能。

适应性时间顺序学分分配的配对权重