This paper proposes a multiagent based bi-level operation framework for the
low-carbon demand management in distribution networks considering the carbon
emission allowance on the demand side. In the upper level, the aggregate load
agents optimize the control signals for various types of loads to maximize the
profits; in the lower level, the distribution network operator makes optimal
dispatching decisions to minimize the operational costs and calculates the
distribution locational marginal price and carbon intensity. The distributed
flexible load agent has only incomplete information of the distribution network
and cooperates with other agents using networked communication. Finally, the
problem is formulated into a networked multi-agent constrained Markov decision
process, which is solved using a safe reinforcement learning algorithm called
consensus multi-agent constrained policy optimization considering the carbon
emission allowance for each agent. Case studies with the IEEE 33-bus and
123-bus distribution network systems demonstrate the effectiveness of the
proposed approach, in terms of satisfying the carbon emission constraint on
demand side, ensuring the safe operation of the distribution network and
preserving privacy of both sides.

该研究论文提出了一个基于多智能体的双层运营框架，用于考虑需求端的碳排放限额的配电网络低碳需求管理，其中上层通过优化控制信号最大化利润，下层通过优化调度决策最小化运营成本并计算配电边际价格和碳强度。通过网络通信与其他智能体合作，分布式柔性负荷智能体利用配电网络的不完全信息。最终，该问题被建模为一个基于网络的多智能体有约束马尔可夫决策过程，并使用一种名为共识多智能体有约束策略优化的安全强化学习算法进行求解，该算法考虑了每个智能体的碳排放限额。在 IEEE 33 节点和 123 节点配电网系统上的案例研究证明了所提方法的有效性，包括满足需求侧的碳排放约束、确保配电网的安全运行和保护双方的隐私。

分布式网络多智能体安全强化学习在低碳需求管理中的应用

Networked Multiagent Safe Reinforcement Learning for Low-carbon Demand  Management in Distribution Network

Collective human knowledge has clearly benefited from the fact that
innovations by individuals are taught to others through communication. Similar
to human social groups, agents in distributed learning systems would likely
benefit from communication to share knowledge and teach skills. The problem of
teaching to improve agent learning has been investigated by prior works, but
these approaches make assumptions that prevent application of teaching to
general multiagent problems, or require domain expertise for problems they can
apply to. This learning to teach problem has inherent complexities related to
measuring long-term impacts of teaching that compound the standard multiagent
coordination challenges. In contrast to existing works, this paper presents the
first general framework and algorithm for intelligent agents to learn to teach
in a multiagent environment. Our algorithm, Learning to Coordinate and Teach
Reinforcement (LeCTR), addresses peer-to-peer teaching in cooperative
multiagent reinforcement learning. Each agent in our approach learns both when
and what to advise, then uses the received advice to improve local learning.
Importantly, these roles are not fixed; these agents learn to assume the role
of student and/or teacher at the appropriate moments, requesting and providing
advice in order to improve teamwide performance and learning. Empirical
comparisons against state-of-the-art teaching methods show that our teaching
agents not only learn significantly faster, but also learn to coordinate in
tasks where existing methods fail.

本文提出了一种新的算法，名为 Learning to Coordinate and Teach Reinforcement（LeCTR），通过在协作多智能体强化学习中使每个代理都学习何时提供何种建议，从而改善整个团队性能和学习效果。实证比较表明，我们的教学代理不仅学习速度更快，而且在现有方法失败的任务中也学会了协作。