Policy Mirror Descent (PMD) is a powerful and theoretically sound methodology
for sequential decision-making. However, it is not directly applicable to
Reinforcement Learning (RL) due to the inaccessibility of explicit action-value
functions. We address this challenge by introducing a novel approach based on
learning a world model of the environment using conditional mean embeddings. We
then leverage the operatorial formulation of RL to express the action-value
function in terms of this quantity in closed form via matrix operations.
Combining these estimators with PMD leads to POWR, a new RL algorithm for which
we prove convergence rates to the global optimum. Preliminary experiments in
finite and infinite state settings support the effectiveness of our method.

通过使用条件均值嵌入学习环境的世界模型，并利用 RL 的操作性表达式进行矩阵运算，结合 Policy Mirror Descent（PMD）估计量，我们提出了一个新的 RL 算法 POWR，证明了其收敛速度达到全局最优。实验结果表明我们的方法在有限和无限状态设置下是有效的。

运算符世界模型用于强化学习

Operator World Models for Reinforcement Learning

Motivated by applications, we consider here new operator theoretic approaches
to Conditional mean embeddings (CME). Our present results combine a spectral
analysis-based optimization scheme with the use of kernels, stochastic
processes, and constructive learning algorithms. For initially given non-linear
data, we consider optimization-based feature selections. This entails the use
of convex sets of positive definite (p.d.) kernels in a construction of optimal
feature selection via regression algorithms from learning models. Thus, with
initial inputs of training data (for a suitable learning algorithm,) each
choice of p.d. kernel $K$ in turn yields a variety of Hilbert spaces and
realizations of features. A novel idea here is that we shall allow an
optimization over selected sets of kernels $K$ from a convex set $C$ of
positive definite kernels $K$. Hence our \textquotedblleft
optimal\textquotedblright{} choices of feature representations will depend on a
secondary optimization over p.d. kernels $K$ within a specified convex set $C$.

运用新的算子理论方法，结合核函数、随机过程和建设性学习算法，我们提出了用于条件均值嵌入的基于谱分析的优化方案，在优化模型特征选择的过程中，利用正定核的凸集。

通过正定核的条件均值嵌入和最优特征选择

Conditional mean embeddings and optimal feature selection via positive  definite kernels

Conditional mean embeddings (CMEs) have proven themselves to be a powerful
tool in many machine learning applications. They allow the efficient
conditioning of probability distributions within the corresponding reproducing
kernel Hilbert spaces (RKHSs) by providing a linear-algebraic relation for the
kernel mean embeddings of the respective joint and conditional probability
distributions. Both centred and uncentred covariance operators have been used
to define CMEs in the existing literature. In this paper, we develop a
mathematically rigorous theory for both variants, discuss the merits and
problems of each, and significantly weaken the conditions for applicability of
CMEs. In the course of this, we demonstrate a beautiful connection to Gaussian
conditioning in Hilbert spaces.

本文通过对中心化和非中心化协方差算子 CME（Conditional mean embeddings） 的理论研究，讨论了 CME 的优点与问题，大幅放宽了 CME 的适用条件。在此过程中，本文演示了关于 Hilbert 空间中的高斯条件的美妙联系。