Robot policies need to adapt to human preferences and/or new environments. Human experts may have the domain knowledge required to help robots achieve this adaptation. However, existing works often require costly offline re-training on human feedback, and those feedback usually need to be frequent and too complex for the humans to reliably provide. To avoid placing undue burden on human experts and allow quick adaptation in critical real-world situations, we propose designing and sparingly presenting easy-to-answer pairwise action preference queries in an online fashion. Our approach designs queries and determines when to present them to maximize the expected value derived from the queries' information. We demonstrate our approach with experiments in simulation, human user studies, and real robot experiments. In these settings, our approach outperforms baseline techniques while presenting fewer queries to human experts. Experiment videos, code and appendices are found at https://sites.google.com/view/onlineactivepreferences.

为了实现机器人在不同环境和人类偏好中的适应性，本研究提出了在线稀疏二元偏好查询的方法，通过设计查询和决定何时呈现查询来最大化查询结果的信息价值，从而使得机器人能够快速适应实际应用场景并减轻人类专家的负担，在模拟，人类用户研究和真实机器人实验中，该方法表现优于传统技术并减少人类专家的查询负担。

在线偏好中的积极奖励学习