For autonomous agents to successfully integrate into human-centered
environments, agents should be able to learn from and adapt to humans in their
native settings. Preference-based reinforcement learning (PbRL) is a promising
approach that learns reward functions from human preferences. This enables RL
agents to adapt their behavior based on human desires. However, humans live in
a world full of diverse information, most of which is not relevant to
completing a particular task. It becomes essential that agents learn to focus
on the subset of task-relevant environment features. Unfortunately, prior work
has largely ignored this aspect; primarily focusing on improving PbRL
algorithms in standard RL environments that are carefully constructed to
contain only task-relevant features. This can result in algorithms that may not
effectively transfer to a more noisy real-world setting. To that end, this work
proposes R2N (Robust-to-Noise), the first PbRL algorithm that leverages
principles of dynamic sparse training to learn robust reward models that can
focus on task-relevant features. We study the effectiveness of R2N in the
Extremely Noisy Environment setting, an RL problem setting where up to 95% of
the state features are irrelevant distractions. In experiments with a simulated
teacher, we demonstrate that R2N can adapt the sparse connectivity of its
neural networks to focus on task-relevant features, enabling R2N to
significantly outperform several state-of-the-art PbRL algorithms in multiple
locomotion and control environments.

为了在人类中心环境中成功融入自主代理，代理应该能够从人类的本地环境中学习和适应。基于偏好的强化学习 (PbRL) 是一种能够从人类偏好中学习奖励函数的有希望的方法，使得强化学习代理能够根据人类的欲望来调整其行为。然而，人类生活在一个充满各种信息的世界中，其中大部分与完成特定任务无关。本工作提出了第一个利用动态稀疏训练原理学习健壮奖励模型并能够专注于任务相关特征的 PbRL 算法 R2N (Robust-to-Noise)。我们在极其嘈杂的环境设置中研究了 R2N 的有效性，该环境中高达 95% 的状态特征都是干扰项。通过与模拟教师的实验，我们证明了 R2N 能够调整其神经网络的稀疏连接性以专注于任务相关特征，在多个运动和控制环境中明显优于几种最先进的 PbRL 算法。