The role of natural language constraints in Safe Multi-agent Reinforcement
Learning (MARL) is crucial, yet often overlooked. While Safe MARL has vast
potential, especially in fields like robotics and autonomous vehicles, its full
potential is limited by the need to define constraints in pre-designed
mathematical terms, which requires extensive domain expertise and reinforcement
learning knowledge, hindering its broader adoption. To address this limitation
and make Safe MARL more accessible and adaptable, we propose a novel approach
named Safe Multi-agent Reinforcement Learning with Natural Language constraints
(SMALL). Our method leverages fine-tuned language models to interpret and
process free-form textual constraints, converting them into semantic embeddings
that capture the essence of prohibited states and behaviours. These embeddings
are then integrated into the multi-agent policy learning process, enabling
agents to learn policies that minimize constraint violations while optimizing
rewards. To evaluate the effectiveness of SMALL, we introduce the LaMaSafe, a
multi-task benchmark designed to assess the performance of multiple agents in
adhering to natural language constraints. Empirical evaluations across various
environments demonstrate that SMALL achieves comparable rewards and
significantly fewer constraint violations, highlighting its effectiveness in
understanding and enforcing natural language constraints.

我们提出了一种名为 SMALL 的新方法，该方法利用经过微调的语言模型来解释和处理自由形式的文本约束，将其转化为捕捉禁止状态和行为本质的语义嵌入，并将其整合到多智能体策略学习过程中，使智能体能够学习最小化约束违规和优化奖励的策略。在多个环境中进行的实证评估显示，SMALL 在理解和执行自然语言约束方面具有显著的效果，实现了可比较的奖励和显著减少的约束违规。

自然语言约束下的安全多智能体强化学习

Safe Multi-agent Reinforcement Learning with Natural Language  Constraints

Safe reinforcement learning (RL) agents accomplish given tasks while adhering
to specific constraints. Employing constraints expressed via
easily-understandable human language offers considerable potential for
real-world applications due to its accessibility and non-reliance on domain
expertise. Previous safe RL methods with natural language constraints typically
adopt a recurrent neural network, which leads to limited capabilities when
dealing with various forms of human language input. Furthermore, these methods
often require a ground-truth cost function, necessitating domain expertise for
the conversion of language constraints into a well-defined cost function that
determines constraint violation. To address these issues, we proposes to use
pre-trained language models (LM) to facilitate RL agents' comprehension of
natural language constraints and allow them to infer costs for safe policy
learning. Through the use of pre-trained LMs and the elimination of the need
for a ground-truth cost, our method enhances safe policy learning under a
diverse set of human-derived free-form natural language constraints.
Experiments on grid-world navigation and robot control show that the proposed
method can achieve strong performance while adhering to given constraints. The
usage of pre-trained LMs allows our method to comprehend complicated
constraints and learn safe policies without the need for ground-truth cost at
any stage of training or evaluation. Extensive ablation studies are conducted
to demonstrate the efficacy of each part of our method.

利用预训练语言模型的先验知识，我们的方法可以在遵守给定约束条件的情况下，理解复杂约束并学习安全策略，而无需任何阶段的真实成本。

使用自由形式的自然语言约束和预训练语言模型的安全强化学习

Safe Reinforcement Learning with Free-form Natural Language Constraints  and Pre-Trained Language Models

We consider the task of text generation in language models with constraints
specified in natural language. To this end, we first create a challenging
benchmark Cognac that provides as input to the model a topic with example text,
along with a constraint on text to be avoided. Unlike prior work, our benchmark
contains knowledge-intensive constraints sourced from databases like Wordnet
and Wikidata, which allows for straightforward evaluation while striking a
balance between broad attribute-level and narrow lexical-level controls. We
find that even state-of-the-art language models like GPT-3 fail often on this
task, and propose a solution to leverage a language model's own internal
knowledge to guide generation. Our method, called CognacGen, first queries the
language model to generate guidance terms for a specified topic or constraint,
and uses the guidance to modify the model's token generation probabilities. We
propose three forms of guidance (binary verifier, top-k tokens, textual
example), and employ prefix-tuning approaches to distill the guidance to tackle
diverse natural language constraints. Through extensive empirical evaluations,
we demonstrate that CognacGen can successfully generalize to unseen
instructions and outperform competitive baselines in generating constraint
conforming text.

本篇论文提出了使用自然语言指定限制条件的语言模型的文本生成任务，并创建了挑战性的基准 Cognac。我们提出了一种名为 CognacGen 的方法，旨在利用语言模型的内部知识来引导生成，它可以成功地推广到未看到的指令，并在生成符合限制的文本方面优于竞争基线。