In this paper, we provide a simple proof from scratch for the convergence of
the Policy Improvement Algorithm(PIA) for a continuous time entropy-regularized
stochastic control problem. Such convergence has been established by
Huang-Wang-Zhou(2023) by using sophisticated PDE estimates for the iterative
PDEs involved in the PIA. Our approach builds on some Feynman-Kac type
probabilistic representation formulae for solutions of PDEs and their
derivatives. Moreover, in the infinite horizon model with a large discount
factor and in the finite horizon model, we obtain the exponential rate of
convergence with similar arguments.

该论文提供了一个简单的证明，从头开始验证了连续时间熵正则化随机控制问题的策略改进算法的收敛性。通过利用 PIA 所涉及的迭代 PDE 中的复杂 PDE 估计，Huang-Wang-Zhou (2023) 已经建立了这样的收敛性。我们的方法基于一些 Feynman-Kac 类型的概率表示公式及其导数的解。此外，在具有较大折扣因子的无穷时间模型和有限时间模型中，我们通过类似的论证得到了指数级收敛速度。

关于策略改进算法的收敛性和收敛速度

On Convergence and Rate of Convergence of Policy Improvement Algorithms

A comparison-based search algorithm lets a user find a target item $t$ in a
database by answering queries of the form, ``Which of items $i$ and $j$ is
closer to $t$?'' Instead of formulating an explicit query (such as one or
several keywords), the user navigates towards the target via a sequence of such
(typically noisy) queries.
We propose a scale-free probabilistic oracle model called $\gamma$-CKL for
such similarity triplets $(i,j;t)$, which generalizes the CKL triplet model
proposed in the literature. The generalization affords independent control over
the discriminating power of the oracle and the dimension of the feature space
containing the items.
We develop a search algorithm with provably exponential rate of convergence
under the $\gamma$-CKL oracle, thanks to a backtracking strategy that deals
with the unavoidable errors in updating the belief region around the target.
We evaluate the performance of the algorithm both over the posited oracle and
over several real-world triplet datasets. We also report on a comprehensive
user study, where human subjects navigate a database of face portraits.

本文提出了一种基于对比的搜索算法和相应的概率预测模型，旨在提高搜索的效率和收敛速度，并进行了一系列的实验来验证其性能。