Energy storage systems (ESS) are pivotal component in the energy market,
serving as both energy suppliers and consumers. ESS operators can reap benefits
from energy arbitrage by optimizing operations of storage equipment. To further
enhance ESS flexibility within the energy market and improve renewable energy
utilization, a heterogeneous photovoltaic-ESS (PV-ESS) is proposed, which
leverages the unique characteristics of battery energy storage (BES) and
hydrogen energy storage (HES). For scheduling tasks of the heterogeneous
PV-ESS, cost description plays a crucial role in guiding operator's strategies
to maximize benefits. We develop a comprehensive cost function that takes into
account degradation, capital, and operation/maintenance costs to reflect
real-world scenarios. Moreover, while numerous methods excel in optimizing ESS
energy arbitrage, they often rely on black-box models with opaque
decision-making processes, limiting practical applicability. To overcome this
limitation and enable transparent scheduling strategies, a prototype-based
policy network with inherent interpretability is introduced. This network
employs human-designed prototypes to guide decision-making by comparing
similarities between prototypical situations and encountered situations, which
allows for naturally explained scheduling strategies. Comparative results
across four distinct cases underscore the effectiveness and practicality of our
proposed pre-hoc interpretable optimization method when contrasted with
black-box models.

能源储存系统是能源市场中至关重要的组成部分，本研究提出了一种异质光伏 - 能源储存系统，利用了电池能量存储和氢能存储的特性，通过开发全面的成本函数和基于人为设计的原型网络，实现了透明的调度策略并取得了令人满意的效果。

优化异构能源存储系统的可解释深度强化学习

Interpretable Deep Reinforcement Learning for Optimizing Heterogeneous  Energy Storage Systems

The optimal dispatch of energy storage systems (ESSs) presents formidable
challenges due to the uncertainty introduced by fluctuations in dynamic prices,
demand consumption, and renewable-based energy generation. By exploiting the
generalization capabilities of deep neural networks (DNNs), deep reinforcement
learning (DRL) algorithms can learn good-quality control models that adaptively
respond to distribution networks' stochastic nature. However, current DRL
algorithms lack the capabilities to enforce operational constraints strictly,
often even providing unfeasible control actions. To address this issue, we
propose a DRL framework that effectively handles continuous action spaces while
strictly enforcing the environments and action space operational constraints
during online operation. Firstly, the proposed framework trains an action-value
function modeled using DNNs. Subsequently, this action-value function is
formulated as a mixed-integer programming (MIP) formulation enabling the
consideration of the environment's operational constraints. Comprehensive
numerical simulations show the superior performance of the proposed MIP-DRL
framework, effectively enforcing all constraints while delivering high-quality
dispatch decisions when compared with state-of-the-art DRL algorithms and the
optimal solution obtained with a perfect forecast of the stochastic variables.

提出了一种利用深度强化学习和混合整数规划的框架，用于优化能量储存系统的调度决策，同时严格执行操作约束，以及在存在不确定性的环境中取得高质量的调度决策。

一种用于最优能量储存系统调度的约束强化深度学习框架

A Constraint Enforcement Deep Reinforcement Learning Framework for  Optimal Energy Storage Systems Dispatch

Future power systems will rely heavily on micro grids with a high share of
decentralised renewable energy sources and energy storage systems. The high
complexity and uncertainty in this context might make conventional power
dispatch strategies infeasible. Reinforcement-learning based (RL) controllers
can address this challenge, however, cannot themselves provide safety
guarantees, preventing their deployment in practice. To overcome this
limitation, we propose a formally validated RL controller for economic
dispatch. We extend conventional constraints by a time-dependent constraint
encoding the islanding contingency. The contingency constraint is computed
using set-based backwards reachability analysis and actions of the RL agent are
verified through a safety layer. Unsafe actions are projected into the safe
action space while leveraging constrained zonotope set representations for
computational efficiency. The developed approach is demonstrated on a
residential use case using real-world measurements.

在未来的电力系统中，将大量依赖于具有高比例的分散式可再生能源和能量存储系统的微电网。出于复杂性和不确定性的原因，在这种情境下的传统能源调度策略可能不可行，利用基于强化学习的控制器可以解决这一挑战，但不能提供安全保障，于是我们提出了一种经过正式验证的基于强化学习的经济调度控制器，来克服这种局限性。