Learning rewards from human behaviour or feedback is a promising approach to aligning AI systems with human values but fails to consistently extract correct reward functions. Interpretability tools could enable users to understand and evaluate possible flaws in learned reward functions. We propose Counterfactual Trajectory Explanations (CTEs) to interpret reward functions in reinforcement learning by contrasting an original with a counterfactual partial trajectory and the rewards they each receive. We derive six quality criteria for CTEs and propose a novel Monte-Carlo-based algorithm for generating CTEs that optimises these quality criteria. Finally, we measure how informative the generated explanations are to a proxy-human model by training it on CTEs. CTEs are demonstrably informative for the proxy-human model, increasing the similarity between its predictions and the reward function on unseen trajectories. Further, it learns to accurately judge differences in rewards between trajectories and generalises to out-of-distribution examples. Although CTEs do not lead to a perfect understanding of the reward, our method, and more generally the adaptation of XAI methods, are presented as a fruitful approach for interpreting learned reward functions.

通过对比原始和反事实的部分轨迹及其各自接收的奖励，我们提出了反事实轨迹解释（CTEs）来解释强化学习中的奖励函数，证明了CTEs对代理人模型具有信息量，使其预测与未见轨迹上的奖励函数更加相似，并且在不同分布的示例中具有泛化能力，从而为解释学习的奖励函数提供了一种有效的方法。

解释学习得到的奖励函数与反事实轨迹