Safe reinforcement learning (RL) agents accomplish given tasks while adhering
to specific constraints. Employing constraints expressed via
easily-understandable human language offers considerable potential for
real-world applications due to its accessibility and non-reliance on domain
expertise. Previous safe RL methods with natural language constraints typically
adopt a recurrent neural network, which leads to limited capabilities when
dealing with various forms of human language input. Furthermore, these methods
often require a ground-truth cost function, necessitating domain expertise for
the conversion of language constraints into a well-defined cost function that
determines constraint violation. To address these issues, we proposes to use
pre-trained language models (LM) to facilitate RL agents' comprehension of
natural language constraints and allow them to infer costs for safe policy
learning. Through the use of pre-trained LMs and the elimination of the need
for a ground-truth cost, our method enhances safe policy learning under a
diverse set of human-derived free-form natural language constraints.
Experiments on grid-world navigation and robot control show that the proposed
method can achieve strong performance while adhering to given constraints. The
usage of pre-trained LMs allows our method to comprehend complicated
constraints and learn safe policies without the need for ground-truth cost at
any stage of training or evaluation. Extensive ablation studies are conducted
to demonstrate the efficacy of each part of our method.

利用预训练语言模型的先验知识，我们的方法可以在遵守给定约束条件的情况下，理解复杂约束并学习安全策略，而无需任何阶段的真实成本。