Active Voltage Control (AVC) on the Power Distribution Networks (PDNs) aims
to stabilize the voltage levels to ensure efficient and reliable operation of
power systems. With the increasing integration of distributed energy resources,
recent efforts have explored employing multi-agent reinforcement learning
(MARL) techniques to realize effective AVC. Existing methods mainly focus on
the acquisition of short-term AVC strategies, i.e., only learning AVC within
the short-term training trajectories of a singular diurnal cycle. However, due
to the dynamic nature of load demands and renewable energy, the operation
states of real-world PDNs may exhibit significant distribution shifts across
varying timescales (e.g., daily and seasonal changes). This can render those
short-term strategies suboptimal or even obsolete when performing continuous
AVC over extended periods. In this paper, we propose a novel temporal
prototype-aware learning method, abbreviated as TPA, to learn time-adaptive AVC
under short-term training trajectories. At the heart of TPA are two
complementary components, namely multi-scale dynamic encoder and temporal
prototype-aware policy, that can be readily incorporated into various MARL
methods. The former component integrates a stacked transformer network to learn
underlying temporal dependencies at different timescales of the PDNs, while the
latter implements a learnable prototype matching mechanism to construct a
dedicated AVC policy that can dynamically adapt to the evolving operation
states. Experimental results on the AVC benchmark with different PDN sizes
demonstrate that the proposed TPA surpasses the state-of-the-art counterparts
not only in terms of control performance but also by offering model
transferability. Our code is available at
this https URL

提出了一种新的时间原型感知学习方法来学习短期训练轨迹下的时变电压控制，该方法在控制性能上不仅超过了现有技术，还提供了模型可转移性。

基于时间原型感知的主动电压控制在电力配电网络上的学习

Temporal Prototype-Aware Learning for Active Voltage Control on Power  Distribution Networks

In this paper, we consider the problem of reference tracking in uncertain
nonlinear systems. A neural State-Space Model (NSSM) is used to approximate the
nonlinear system, where a deep encoder network learns the nonlinearity from
data, and a state-space component captures the temporal relationship. This
transforms the nonlinear system into a linear system in a latent space,
enabling the application of model predictive control (MPC) to determine
effective control actions. Our objective is to design the optimal controller
using limited data from the \textit{target system} (the system of interest). To
this end, we employ an implicit model-agnostic meta-learning (iMAML) framework
that leverages information from \textit{source systems} (systems that share
similarities with the target system) to expedite training in the target system
and enhance its control performance. The framework consists of two phases: the
(offine) meta-training phase learns a aggregated NSSM using data from source
systems, and the (online) meta-inference phase quickly adapts this aggregated
model to the target system using only a few data points and few online training
iterations, based on local loss function gradients. The iMAML algorithm
exploits the implicit function theorem to exactly compute the gradient during
training, without relying on the entire optimization path. By focusing solely
on the optimal solution, rather than the path, we can meta-train with less
storage complexity and fewer approximations than other contemporary
meta-learning algorithms. We demonstrate through numerical examples that our
proposed method can yield accurate predictive models by adaptation, resulting
in a downstream MPC that outperforms several baselines.

使用神经状态空间模型和隐式模型无关元学习方法解决非线性系统中的参考跟踪问题，并通过模型预测控制提高控制性能。

基于元学习的快速适应神经预测模型的不确定非线性系统 MPC

MPC of Uncertain Nonlinear Systems with Meta-Learning for Fast  Adaptation of Neural Predictive Models

Model-based reinforcement learning (MBRL) has been shown to be a powerful
framework for data-efficiently learning control of continuous tasks. Recent
work in MBRL has mostly focused on using more advanced function approximators
and planning schemes, with little development of the general framework. In this
paper, we identify a fundamental issue of the standard MBRL framework -- what
we call the objective mismatch issue. Objective mismatch arises when one
objective is optimized in the hope that a second, often uncorrelated, metric
will also be optimized. In the context of MBRL, we characterize the objective
mismatch between training the forward dynamics model w.r.t.~the likelihood of
the one-step ahead prediction, and the overall goal of improving performance on
a downstream control task. For example, this issue can emerge with the
realization that dynamics models effective for a specific task do not
necessarily need to be globally accurate, and vice versa globally accurate
models might not be sufficiently accurate locally to obtain good control
performance on a specific task. In our experiments, we study this objective
mismatch issue and demonstrate that the likelihood of one-step ahead
predictions is not always correlated with control performance. This observation
highlights a critical limitation in the MBRL framework which will require
further research to be fully understood and addressed. We propose an initial
method to mitigate the mismatch issue by re-weighting dynamics model training.
Building on it, we conclude with a discussion about other potential directions
of research for addressing this issue.

本文研究模型驱动强化学习中的目标不匹配问题，通过探讨训练前向动力学模型的似然性和提高下游控制任务绩效之间的关系，发现单纯遵照似然性训练不一定能提高控制性能，提出一种缓解目标不匹配问题的方法并探讨了其他潜在的解决方案。

基于模型的强化学习中的目标不匹配

Objective Mismatch in Model-based Reinforcement Learning

Domain knowledge is crucial for effective performance in autonomous control
systems. Typically, human effort is required to encode this knowledge into a
control algorithm. In this paper, we present an approach to language grounding
which automatically interprets text in the context of a complex control
application, such as a game, and uses domain knowledge extracted from the text
to improve control performance. Both text analysis and control strategies are
learned jointly using only a feedback signal inherent to the application. To
effectively leverage textual information, our method automatically extracts the
text segment most relevant to the current game state, and labels it with a
task-centric predicate structure. This labeled text is then used to bias an
action selection policy for the game, guiding it towards promising regions of
the action space. We encode our model for text analysis and game playing in a
multi-layer neural network, representing linguistic decisions via latent
variables in the hidden layers, and game action quality via the output layer.
Operating within the Monte-Carlo Search framework, we estimate model parameters
using feedback from simulated games. We apply our approach to the complex
strategy game Civilization II using the official game manual as the text guide.
Our results show that a linguistically-informed game-playing agent
significantly outperforms its language-unaware counterpart, yielding a 34%
absolute improvement and winning over 65% of games when playing against the
built-in AI of Civilization.

本文介绍了一种通过语言基础知识自动为控制算法提供指导的方法，并使用多层神经网络表示文本分析与游戏操作决策，实验证明这种基于语言知识的游戏代理系统可以显著提高控制性能和获胜概率。