This article proposes a model-based deep reinforcement learning (DRL) method
to design emergency control strategies for short-term voltage stability
problems in power systems. Recent advances show promising results in model-free
DRL-based methods for power systems, but model-free methods suffer from poor
sample efficiency and training time, both critical for making state-of-the-art
DRL algorithms practically applicable. DRL-agent learns an optimal policy via a
trial-and-error method while interacting with the real-world environment. And
it is desirable to minimize the direct interaction of the DRL agent with the
real-world power grid due to its safety-critical nature. Additionally,
state-of-the-art DRL-based policies are mostly trained using a physics-based
grid simulator where dynamic simulation is computationally intensive, lowering
the training efficiency. We propose a novel model-based-DRL framework where a
deep neural network (DNN)-based dynamic surrogate model, instead of a
real-world power-grid or physics-based simulation, is utilized with the policy
learning framework, making the process faster and sample efficient. However,
stabilizing model-based DRL is challenging because of the complex system
dynamics of large-scale power systems. We solved these issues by incorporating
imitation learning to have a warm start in policy learning, reward-shaping, and
multi-step surrogate loss. Finally, we achieved 97.5% sample efficiency and
87.7% training efficiency for an application to the IEEE 300-bus test system.

提出了一种基于模型的深度强化学习方法，用于设计短期电压稳定性问题的紧急控制策略，并使用基于深度神经网络 的动态代理模型进行训练，最终在 IEEE 300 总线测试系统上实现了 97.5% 的样本效率和 87.7% 的训练效率。