In safety-critical applications, autonomous agents may need to learn in an
environment where mistakes can be very costly. In such settings, the agent
needs to behave safely not only after but also while learning. To achieve this,
existing safe reinforcement learning methods make an agent rely on priors that
let it avoid dangerous situations during exploration with high probability, but
both the probabilistic guarantees and the smoothness assumptions inherent in
the priors are not viable in many scenarios of interest such as autonomous
driving. This paper presents an alternative approach inspired by human
teaching, where an agent learns under the supervision of an automatic
instructor that saves the agent from violating constraints during learning. In
this model, we introduce the monitor that neither needs to know how to do well
at the task the agent is learning nor needs to know how the environment works.
Instead, it has a library of reset controllers that it activates when the agent
starts behaving dangerously, preventing it from doing damage. Crucially, the
choices of which reset controller to apply in which situation affect the speed
of agent learning. Based on observing agents' progress, the teacher itself
learns a policy for choosing the reset controllers, a curriculum, to optimize
the agent's final policy reward. Our experiments use this framework in two
environments to induce curricula for safe and efficient learning.

本文提出一种受人类教学启发的替代方法，即代理在自动指导监督下学习，其中引入了监视器来防止其在学习过程中违反约束条件。

通过课程引导实现安全强化学习

Safe Reinforcement Learning via Curriculum Induction

Machines, not humans, are the world's dominant knowledge accumulators but
humans remain the dominant decision makers. Interpreting and disseminating the
knowledge accumulated by machines requires expertise, time, and is prone to
failure. The problem of how best to convey accumulated knowledge from computers
to humans is a critical bottleneck in the broader application of machine
learning. We propose an approach based on human teaching where the problem is
formalized as selecting a small subset of the data that will, with high
probability, lead the human user to the correct inference. This approach,
though successful for modeling human learning in simple laboratory experiments,
has failed to achieve broader relevance due to challenges in formulating
general and scalable algorithms. We propose general-purpose teaching via
pseudo-marginal sampling and demonstrate the algorithm by teaching topic
models. Simulation results show our sampling-based approach: effectively
approximates the probability where ground-truth is possible via enumeration,
results in data that are markedly different from those expected by random
sampling, and speeds learning especially for small amounts of data. Application
to movie synopsis data illustrates differences between teaching and random
sampling for teaching distributions and specific topics, and demonstrates gains
in scalability and applicability to real-world problems.

通过基于人类教学的伪边际抽样实现机器学习的普适性教学方法，成功地在主题模型上进行训练，并在电影简介数据中验证了该方法的优势。