We study a Federated Reinforcement Learning (FedRL) problem with constraint
heterogeneity. In our setting, we aim to solve a reinforcement learning problem
with multiple constraints while $N$ training agents are located in $N$
different environments with limited access to the constraint signals and they
are expected to collaboratively learn a policy satisfying all constraint
signals. Such learning problems are prevalent in scenarios of Large Language
Model (LLM) fine-tuning and healthcare applications. To solve the problem, we
propose federated primal-dual policy optimization methods based on traditional
policy gradient methods. Specifically, we introduce $N$ local Lagrange
functions for agents to perform local policy updates, and these agents are then
scheduled to periodically communicate on their local policies. Taking natural
policy gradient (NPG) and proximal policy optimization (PPO) as policy
optimization methods, we mainly focus on two instances of our algorithms, ie,
{FedNPG} and {FedPPO}. We show that FedNPG achieves global convergence with an
$\tilde{O}(1/\sqrt{T})$ rate, and FedPPO efficiently solves complicated
learning tasks with the use of deep neural networks.

我们研究了具有约束异构性的联邦强化学习（FedRL）问题，通过多个智能体在不同环境下协同学习，解决具有多个约束的强化学习问题，并提出了基于传统策略梯度方法的联邦原始 - 对偶策略优化方法，其中我们关注的两种算法为 FedNPG 和 FedPPO，通过使用深度神经网络，FedPPO 有效地解决了复杂的学习任务。