Training automated agents to complete complex tasks in interactive
environments is challenging: reinforcement learning requires careful
hand-engineering of reward functions, imitation learning requires specialized
infrastructure and access to a human expert, and learning from intermediate
forms of supervision (like binary preferences) is time-consuming and extracts
little information from each human intervention. Can we overcome these
challenges by building agents that learn from rich, interactive feedback
instead? We propose a new supervision paradigm for interactive learning based
on "teachable" decision-making systems that learn from structured advice
provided by an external teacher. We begin by formalizing a class of
human-in-the-loop decision making problems in which multiple forms of
teacher-provided advice are available to a learner. We then describe a simple
learning algorithm for these problems that first learns to interpret advice,
then learns from advice to complete tasks even in the absence of human
supervision. In puzzle-solving, navigation, and locomotion domains, we show
that agents that learn from advice can acquire new skills with significantly
less human supervision than standard reinforcement learning algorithms and
often less than imitation learning.

本研究提出了一种基于 “可教学” 的决策系统的交互式学习监督范式，能够通过学习外部教师提供的结构化建议，解决复杂任务的学习难度，对拼图、导航和运动等各种任务需求的人工干预也相对较少。

透过指引提炼的可教授强化学习

Teachable Reinforcement Learning via Advice Distillation

The classification and regression head are both indispensable components to
build up a dense object detector, which are usually supervised by the same
training samples and thus expected to have consistency with each other for
detecting objects accurately in the detection pipeline. In this paper, we break
the convention of the same training samples for these two heads in dense
detectors and explore a novel supervisory paradigm, termed as Mutual
Supervision (MuSu), to respectively and mutually assign training samples for
the classification and regression head to ensure this consistency. MuSu defines
training samples for the regression head mainly based on classification
predicting scores and in turn, defines samples for the classification head
based on localization scores from the regression head. Experimental results
show that the convergence of detectors trained by this mutual supervision is
guaranteed and the effectiveness of the proposed method is verified on the
challenging MS COCO benchmark. We also find that tiling more anchors at the
same location benefits detectors and leads to further improvements under this
training scheme. We hope this work can inspire further researches on the
interaction of the classification and regression task in detection and the
supervision paradigm for detectors, especially separately for these two heads.

本文提出了一种新的监督范式，称为相互监视（MuSu），为密集探测器中的分类和回归头分别和相互分配训练样本，以确保对检测管道中的物体进行准确检测的一致性。实验结果表明，使用这种相互监视进行训练可以保证检测器的收敛性，并且在具有挑战性的 MS COCO 基准测试中验证了所提出的方法的有效性。