Although deep reinforcement learning has demonstrated impressive achievements
in controlling various autonomous systems, e.g., autonomous vehicles or
humanoid robots, its inherent reliance on random exploration raises safety
concerns in their real-world applications. To improve system safety during the
learning process, a variety of Safe Reinforcement Learning (SRL) algorithms
have been proposed, which usually incorporate safety constraints within the
Constrained Markov Decision Process (CMDP) framework. However, the efficacy of
these SRL algorithms often relies on accurate function approximations, a task
that is notably challenging to accomplish in the early learning stages due to
data insufficiency. To address this problem, we introduce a Genralizable Safety
enhancer (GenSafe) in this work. Leveraging model order reduction techniques,
we first construct a Reduced Order Markov Decision Process (ROMDP) as a
low-dimensional proxy for the original cost function in CMDP. Then, by solving
ROMDP-based constraints that are reformulated from the original cost
constraints, the proposed GenSafe refines the actions taken by the agent to
enhance the possibility of constraint satisfaction. Essentially, GenSafe acts
as an additional safety layer for SRL algorithms, offering broad compatibility
across diverse SRL approaches. The performance of GenSafe is examined on
multiple SRL benchmark problems. The results show that, it is not only able to
improve the safety performance, especially in the early learning phases, but
also to maintain the task performance at a satisfactory level.

为了提高深度强化学习中系统的安全性，在这项工作中引入了一个名为 GenSafe 的通用安全增强器，通过模型降阶技术构建了一个低维度的 Proxy 来改善代理行为，从而提供了跨多种 SRL 方法的广泛兼容性，它不仅能够改善安全性能，特别是在早期学习阶段，还能够维持任务性能在一个令人满意的水平。