In preference-based reinforcement learning (PbRL), a reward function is
learned from a type of human feedback called preference. To expedite preference
collection, recent works have leveraged \emph{offline preferences}, which are
preferences collected for some offline data. In this scenario, the learned
reward function is fitted on the offline data. If a learning agent exhibits
behaviors that do not overlap with the offline data, the learned reward
function may encounter generalizability issues. To address this problem, the
present study introduces a framework that consolidates offline preferences and
\emph{virtual preferences} for PbRL, which are comparisons between the agent's
behaviors and the offline data. Critically, the reward function can track the
agent's behaviors using the virtual preferences, thereby offering well-aligned
guidance to the agent. Through experiments on continuous control tasks, this
study demonstrates the effectiveness of incorporating the virtual preferences
in PbRL.

通过整合离线偏好和虚拟偏好，将离线数据和学习代理的行为进行比较，从而加强了基于偏好的强化学习中的奖励函数对代理行为的指导，提高了其泛化能力。