Offline reinforcement learning has become one of the most practical RL
settings. A recent success story has been RLHF, offline preference-based RL
(PBRL) with preference from humans. However, most existing works on offline RL
focus on the standard setting with scalar reward feedback. It remains unknown
how to universally transfer the existing rich understanding of offline RL from
the reward-based to the preference-based setting. In this work, we propose a
general framework to bridge this gap. Our key insight is transforming
preference feedback to scalar rewards via optimal reward labeling (ORL), and
then any reward-based offline RL algorithms can be applied to the dataset with
the reward labels. We theoretically show the connection between several recent
PBRL techniques and our framework combined with specific offline RL algorithms
in terms of how they utilize the preference signals. By combining reward
labeling with different algorithms, our framework can lead to new and
potentially more efficient offline PBRL algorithms. We empirically test our
framework on preference datasets based on the standard D4RL benchmark. When
combined with a variety of efficient reward-based offline RL algorithms, the
learning result achieved under our framework is comparable to training the same
algorithm on the dataset with actual rewards in many cases and better than the
recent PBRL baselines in most cases.

提出了一个通用框架来连接偏好反馈和标量奖励，使得现有的离线 RL 算法能够适应偏好反馈，实验证明该框架加上不同算法可以获得与实际奖励训练相媲美甚至优于离线 PBRL 算法的学习效果。