This paper presents a novel safe reinforcement learning algorithm for
strategic bidding of Virtual Power Plants (VPPs) in day-ahead electricity
markets. The proposed algorithm utilizes the Deep Deterministic Policy Gradient
(DDPG) method to learn competitive bidding policies without requiring an
accurate market model. Furthermore, to account for the complex internal
physical constraints of VPPs we introduce two enhancements to the DDPG method.
Firstly, a projection-based safety shield that restricts the agent's actions to
the feasible space defined by the non-linear power flow equations and operating
constraints of distributed energy resources is derived. Secondly, a penalty for
the shield activation in the reward function that incentivizes the agent to
learn a safer policy is introduced. A case study based on the IEEE 13-bus
network demonstrates the effectiveness of the proposed approach in enabling the
agent to learn a highly competitive, safe strategic policy.

该论文提出了一种新颖的安全强化学习算法，用于虚拟电力厂在日前电力市场上的战略竞标，同时考虑了虚拟电力厂内部复杂的物理约束，并结合 DDPG 方法设计了投影安全屏障和奖励机制，最终在 IEEE 13-bus 网络上展示了提出方法的有效性。

面向日前市场的虚拟电力厂战略出价安全强化学习

Safe Reinforcement Learning for Strategic Bidding of Virtual Power  Plants in Day-Ahead Markets

Virtual power plants and load aggregation are becoming increasingly common.
There, one regulates the aggregate power output of an ensemble of distributed
energy resources (DERs). Marecek et al. [Automatica, Volume 147, January 2023,
110743, arXiv:2110.03001] recently suggested that long-term averages of prices
or incentives offered should exist and be independent of the initial states of
the operators of the DER, the aggregator, and the power grid. This can be seen
as predictability, which underlies fairness. Unfortunately, the existence of
such averages cannot be guaranteed with many traditional regulators, including
the proportional-integral (PI) regulator with or without deadband. Here, we
consider the effects of losses in the alternating current model and the
deadband in the controller. This yields a non-linear dynamical system (due to
the non-linear losses) exhibiting discontinuities (due to the deadband). We
show that Filippov invariant measures enable reasoning about predictability and
fairness while considering non-linearity of the alternating-current model and
deadband.

该文章提出使用 Filippov 不变量度量交流模型中的损失和控制器死区对可预测性和公平性的影响。