Preference-Based reinforcement learning (PBRL) learns directly from the preferences of human teachers regarding agent behaviors without needing meticulously designed reward functions. However, existing PBRL methods often learn primarily from explicit preferences, neglecting the possibility that teachers may choose equal preferences. This neglect may hinder the understanding of the agent regarding the task perspective of the teacher, leading to the loss of important information. To address this issue, we introduce the Equal Preference Learning Task, which optimizes the neural network by promoting similar reward predictions when the behaviors of two agents are labeled as equal preferences. Building on this task, we propose a novel PBRL method, Multi-Type Preference Learning (MTPL), which allows simultaneous learning from equal preferences while leveraging existing methods for learning from explicit preferences. To validate our approach, we design experiments applying MTPL to four existing state-of-the-art baselines across ten locomotion and robotic manipulation tasks in the DeepMind Control Suite. The experimental results indicate that simultaneous learning from both equal and explicit preferences enables the PBRL method to more comprehensively understand the feedback from teachers, thereby enhancing feedback efficiency.

本研究解决了现有偏好基础强化学习(PBRL)方法忽视教师可能选择均等偏好的问题，导致对任务理解的不足。通过引入均等偏好学习任务并提出新方法多类型偏好学习（MTPL），同时从均等偏好和显性偏好中学习，从而提高对教师反馈的理解和反馈效率。实验结果表明，MTPL使PBRL在多个人类反馈任务中表现出更强的学习能力。

多类型偏好学习：利用均等偏好的偏好基础强化学习