Training automated agents to complete complex tasks in interactive
environments is challenging: reinforcement learning requires careful
hand-engineering of reward functions, imitation learning requires specialized
infrastructure and access to a human expert, and learning from intermediate
forms of supervision (like binary preferences) is time-consuming and extracts
little information from each human intervention. Can we overcome these
challenges by building agents that learn from rich, interactive feedback
instead? We propose a new supervision paradigm for interactive learning based
on "teachable" decision-making systems that learn from structured advice
provided by an external teacher. We begin by formalizing a class of
human-in-the-loop decision making problems in which multiple forms of
teacher-provided advice are available to a learner. We then describe a simple
learning algorithm for these problems that first learns to interpret advice,
then learns from advice to complete tasks even in the absence of human
supervision. In puzzle-solving, navigation, and locomotion domains, we show
that agents that learn from advice can acquire new skills with significantly
less human supervision than standard reinforcement learning algorithms and
often less than imitation learning.

本研究提出了一种基于 “可教学” 的决策系统的交互式学习监督范式，能够通过学习外部教师提供的结构化建议，解决复杂任务的学习难度，对拼图、导航和运动等各种任务需求的人工干预也相对较少。