Human-in-the-loop reinforcement learning (HRL) allows the training of agents
through various interfaces, even for non-expert humans. Recently,
preference-based methods (PBRL), where the human has to give his preference
over two trajectories, increased in popularity since they allow training in
domains where more direct feedback is hard to formulate. However, the current
PBRL methods have limitations and do not provide humans with an expressive
interface for giving feedback. With this work, we propose a new
preference-based learning method that provides humans with a more expressive
interface to provide their preference over trajectories and a factual
explanation (or annotation of why they have this preference). These
explanations allow the human to explain what parts of the trajectory are most
relevant for the preference. We allow the expression of the explanations over
individual trajectory steps. We evaluate our method in various simulations
using a simulated human oracle (with realistic restrictions), and our results
show that our extended feedback can improve the speed of learning. Code & data:
github.com/under-rewiev

人在环路强化学习中提供了一种可以让非专家用户通过各种界面进行训练的方法。我们提出了一种新的基于偏好的学习方法，为人们提供了更富表现力的界面，以便其表达对轨迹的偏好，并提供了解释以提高学习速度。