Safe reinforcement learning (RL) focuses on training reward-maximizing agents
subject to pre-defined safety constraints. Yet, learning versatile safe
policies that can adapt to varying safety constraint requirements during
deployment without retraining remains a largely unexplored and challenging
area. In this work, we formulate the versatile safe RL problem and consider two
primary requirements: training efficiency and zero-shot adaptation capability.
To address them, we introduce the Conditioned Constrained Policy Optimization
(CCPO) framework, consisting of two key modules: (1) Versatile Value Estimation
(VVE) for approximating value functions under unseen threshold conditions, and
(2) Conditioned Variational Inference (CVI) for encoding arbitrary constraint
thresholds during policy optimization. Our extensive experiments demonstrate
that CCPO outperforms the baselines in terms of safety and task performance
while preserving zero-shot adaptation capabilities to different constraint
thresholds data-efficiently. This makes our approach suitable for real-world
dynamic applications.

我们提出了一种适用于实际动态应用的 Conditioned Constrained Policy Optimization (CCPO) 框架，通过引入 Versatile Value Estimation (VVE) 和 Conditioned Variational Inference (CVI) 两个关键模块，在训练效率和零 - shot 适应能力方面超过基准，同时在安全性和任务性能方面保持一个高水平。