In this paper, we provide a simple proof from scratch for the convergence of
the Policy Improvement Algorithm(PIA) for a continuous time entropy-regularized
stochastic control problem. Such convergence has been established by
Huang-Wang-Zhou(2023) by using sophisticated PDE estimates for the iterative
PDEs involved in the PIA. Our approach builds on some Feynman-Kac type
probabilistic representation formulae for solutions of PDEs and their
derivatives. Moreover, in the infinite horizon model with a large discount
factor and in the finite horizon model, we obtain the exponential rate of
convergence with similar arguments.

该论文提供了一个简单的证明，从头开始验证了连续时间熵正则化随机控制问题的策略改进算法的收敛性。通过利用 PIA 所涉及的迭代 PDE 中的复杂 PDE 估计，Huang-Wang-Zhou (2023) 已经建立了这样的收敛性。我们的方法基于一些 Feynman-Kac 类型的概率表示公式及其导数的解。此外，在具有较大折扣因子的无穷时间模型和有限时间模型中，我们通过类似的论证得到了指数级收敛速度。