Exploration and reward specification are fundamental and intertwined challenges for reinforcement learning. Solving sequential decision-making tasks requiring expansive exploration requires either careful design of reward functions or the use of novelty-seeking exploration bonuses. Human supervisors can provide effective guidance in the loop to direct the exploration process, but prior methods to leverage this guidance require constant synchronous high-quality human feedback, which is expensive and impractical to obtain. In this work, we present a technique called Human Guided Exploration (HuGE), which uses low-quality feedback from non-expert users that may be sporadic, asynchronous, and noisy. HuGE guides exploration for reinforcement learning not only in simulation but also in the real world, all without meticulous reward specification. The key concept involves bifurcating human feedback and policy learning: human feedback steers exploration, while self-supervised learning from the exploration data yields unbiased policies. This procedure can leverage noisy, asynchronous human feedback to learn policies with no hand-crafted reward design or exploration bonuses. HuGE is able to learn a variety of challenging multi-stage robotic navigation and manipulation tasks in simulation using crowdsourced feedback from non-expert users. Moreover, this paradigm can be scaled to learning directly on real-world robots, using occasional, asynchronous feedback from human supervisors.

通过非专业用户提供的低质量、非同步和嘈杂的反馈，人类引导探索（HuGE）技术在强化学习中引导探索，无需精心设计奖励规范。这种方法通过分叉人类反馈和策略学习来实现：人类反馈引导探索，而来自探索数据的自我监督学习产生无偏的策略。HuGE能够借助非专业用户的众包反馈在模拟环境中学习各种具有挑战性的多阶段机器人导航和操纵任务。此外，这个方法还可以通过来自人类监督员的偶尔异步反馈来直接在真实世界中学习。

目标的脚印：从人-环交互反馈中获得目标条件的探索