Safety controllers is widely used to achieve safe reinforcement learning.
Most methods that apply a safety controller are using handcrafted safety
constraints to construct the safety controller. However, when the environment
dynamics are sophisticated, handcrafted safety constraints become unavailable.
Therefore, it worth to research on constructing safety controllers by learning
algorithms. We propose a three-stage architecture for safe reinforcement
learning, namely TU-Recovery Architecture. A safety critic and a recovery
policy is learned before task training. They form a safety controller to ensure
safety in task training. Then a phenomenon induced by disagreement between task
policy and recovery policy, called adversarial phenomenon, which reduces
learning efficiency and model performance, is described. Auxiliary reward is
proposed to mitigate adversarial phenomenon, while help the task policy to
learn to recover from high-risk states. A series of experiments are conducted
in a robot navigation environment. Experiments demonstrate that TU-Recovery
outperforms unconstrained counterpart in both reward gaining and constraint
violations during task training, and auxiliary reward further improve
TU-Recovery in reward-to-cost ratio by significantly reduce constraint
violations.

通过学习算法构建安全控制器以实现安全强化学习，提出了三阶段的架构 TU-Recovery Architecture，并通过辅助奖励来缓解对抗现象，以提高安全控制器在任务训练中的回报率和约束违规率。