Reference-model-assisted deep reinforcement learning (DRL) for inverter-based
Volt-Var Control (IB-VVC) in active distribution networks is proposed. We
investigate that a large action space increases the learning difficulties of
DRL and degrades the optimization performance in the process of generating data
and training neural networks. To reduce the action space of DRL, we design a
reference-model-assisted DRL approach. We introduce definitions of the
reference model, reference-model-based optimization, and reference actions. The
reference-model-assisted DRL learns the residual actions between the reference
actions and optimal actions, rather than learning the optimal actions directly.
Since the residual actions are considerably smaller than the optimal actions
for a reference model, we can design a smaller action space for the
reference-model-assisted DRL. It reduces the learning difficulties of DRL and
optimises the performance of the reference-model-assisted DRL approach. It is
noteworthy that the reference-model-assisted DRL approach is compatible with
any policy gradient DRL algorithms for continuous action problems. This work
takes the soft actor-critic algorithm as an example and designs a
reference-model-assisted soft actor-critic algorithm. Simulations show that 1)
large action space degrades the performance of DRL in the whole training stage,
and 2) reference-model-assisted DRL requires fewer iteration times and returns
a better optimization performance.

提出基于参考模型辅助的深度强化学习方法，通过学习参考动作和最优动作之间的残留动作，缩小动作空间，优化逆变器调压控制在主动配电网中的应用性能。

基于参考模型的深度强化学习在逆变器电压 - 无功控制中的动作空间缩减

Reducing Action Space: Reference-Model-Assisted Deep Reinforcement Learning for Inverter-based Volt-Var Control

The distributed Volt/Var control (VVC) methods have been widely studied for
active distribution networks(ADNs), which is based on perfect model and
real-time P2P communication. However, the model is always incomplete with
significant parameter errors and such P2P communication system is hard to
maintain. In this paper, we propose an online multi-agent reinforcement
learning and decentralized control framework (OLDC) for VVC. In this framework,
the VVC problem is formulated as a constrained Markov game and we propose a
novel multi-agent constrained soft actor-critic (MACSAC) reinforcement learning
algorithm. MACSAC is used to train the control agents online, so the accurate
ADN model is no longer needed. Then, the trained agents can realize
decentralized optimal control using local measurements without real-time P2P
communication. The OLDC with MACSAC has shown extraordinary flexibility,
efficiency and robustness to various computing and communication conditions.
Numerical simulations on IEEE test cases not only demonstrate that the proposed
MACSAC outperforms the state-of-art learning algorithms, but also support the
superiority of our OLDC framework in the online application.

本文提出了一种在线多智能体强化学习和分散式控制框架以解决基于分布式 Volt/Var 控制（VVC）方法所面临的完整性和通信成本问题。在该框架中，通过采用多智能体约束软演员 - 评论家（MACSAC）强化学习算法进行 VVC 管理，不再需要事先准确的 ADN 模型，从而实现分散式优化控制和有效的通信处理。该 OLDC 框架的数值模拟结果表明，它具有出色的灵活性和适应性，可用于各种计算和通信条件下，在线应用中取得更好的控制效果。

基于在线多智能体强化学习的分散式基于逆变器的电压 - 无功控制

Online Multi-agent Reinforcement Learning for Decentralized  Inverter-based Volt-VAR Control

Model-based Vol/VAR optimization method is widely used to eliminate voltage
violations and reduce network losses. However, the parameters of active
distribution networks(ADNs) are not onsite identified, so significant errors
may be involved in the model and make the model-based method infeasible. To
cope with this critical issue, we propose a novel two-stage deep reinforcement
learning (DRL) method to improve the voltage profile by regulating
inverter-based energy resources, which consists of offline stage and online
stage. In the offline stage, a highly efficient adversarial reinforcement
learning algorithm is developed to train an offline agent robust to the model
mismatch. In the sequential online stage, we transfer the offline agent safely
as the online agent to perform continuous learning and controlling online with
significantly improved safety and efficiency. Numerical simulations on IEEE
test cases not only demonstrate that the proposed adversarial reinforcement
learning algorithm outperforms the state-of-art algorithm, but also show that
our proposed two-stage method achieves much better performance than the
existing DRL based methods in the online application.

使用两阶段深度强化学习方法来优化基于逆变器能源的电压调节，通过一个高效的对抗性强化学习算法，训练出离线代理程序，然后将其安全地转移到在线代理程序中，在线学习和控制具有显著的安全性和效率，从而比现有机器学习方法取得更好性能。