Reinforcement learning algorithms discover policies that maximize reward, but
do not necessarily guarantee safety during learning or execution phases. We
introduce a new approach to learn optimal policies while enforcing properties
expressed in temporal logic. To this end, given the temporal logic
specification that is to be obeyed by the learning system, we propose to
synthesize a reactive system called a shield. The shield is introduced in the
traditional learning process in two alternative ways, depending on the location
at which the shield is implemented. In the first one, the shield acts each time
the learning agent is about to make a decision and provides a list of safe
actions. In the second way, the shield is introduced after the learning agent.
The shield monitors the actions from the learner and corrects them only if the
chosen action causes a violation of the specification. We discuss which
requirements a shield must meet to preserve the convergence guarantees of the
learner. Finally, we demonstrate the versatility of our approach on several
challenging reinforcement learning scenarios.

该论文介绍了一种基于时间逻辑表达的安全性要求，用反应系统来指导强化学习算法的政策优化过程，保证学习和执行阶段的安全性。实验表明该方法在多种场景下具有通用性。