We study the global convergence of a Fisher-Rao policy gradient flow for infinite-horizon entropy-regularised Markov decision processes with Polish state and action space. The flow is a continuous-time analogue of a policy mirror descent method. We establish the global well-posedness of the gradient flow and demonstrate its exponential convergence to the optimal policy. Moreover, we prove the flow is stable with respect to gradient evaluation, offering insights into the performance of a natural policy gradient flow with log-linear policy parameterisation. To overcome challenges stemming from the lack of the convexity of the objective function and the discontinuity arising from the entropy regulariser, we leverage the performance difference lemma and the duality relationship between the gradient and mirror descent flows.

我们研究了具有Polish状态和动作空间的无限时段熵正则化马尔可夫决策过程的Fisher-Rao策略梯度流的全球收敛性。该流是策略镜像下降方法的连续时间模拟。我们建立了梯度流的全球适定性，并证明其指数级收敛到最优策略。此外，我们证明了该流在梯度评估方面的稳定性，从而揭示了以对数线性策略参数化的自然策略梯度流的性能。为了克服客观函数的非凸性和由熵正则化引起的不连续性引起的挑战，我们利用了性能差分引理和梯度与镜像下降流之间的对偶关系。

波兰空间中熵正则化的马尔可夫决策过程的费歇-劳梯度流