Maintaining the stability of the modern power grid is becoming increasingly
difficult due to fluctuating power consumption, unstable power supply coming
from renewable energies, and unpredictable accidents such as man-made and
natural disasters. As the operation on the power grid must consider its impact
on future stability, reinforcement learning (RL) has been employed to provide
sequential decision-making in power grid management. However, existing methods
have not considered the environmental constraints. As a result, the learned
policy has risk of selecting actions that violate the constraints in
emergencies, which will escalate the issue of overloaded power lines and lead
to large-scale blackouts. In this work, we propose a novel method for this
problem, which builds on top of the search-based planning algorithm. At the
planning stage, the search space is limited to the action set produced by the
policy. The selected action strictly follows the constraints by testing its
outcome with the simulation function provided by the system. At the learning
stage, to address the problem that gradients cannot be propagated to the
policy, we introduce Evolutionary Strategies (ES) with black-box policy
optimization to improve the policy directly, maximizing the returns of the long
run. In NeurIPS 2020 Learning to Run Power Network (L2RPN) competition, our
solution safely managed the power grid and ranked first in both tracks.

本文提出了一种基于搜索规划算法的新方法解决强化学习中存在的环境约束问题，同时采用黑盒策略优化的进化策略来训练策略直接优化。在 NeurIPS L2RPN 竞赛中，我们的解决方案在两个轨道中均名列第一，能够有效管理电网并确保其地安全性。