Building upon the formalism of regularized Markov decision processes, we study the effect of Kullback-Leibler (KL) and entropy regularization in reinforcement learning. Through an equivalent formulation of the related approximate dynamic programming (ADP) scheme, we show that a KL penalty amounts to averaging q-values. This equivalence allows drawing connections between a priori disconnected methods from the literature, and proving that a KL regularization indeed leads to averaging errors made at each iteration of value function update. With the proposed theoretical analysis, we also study the interplay between KL and entropy regularization. When the considered ADP scheme is combined with neural-network-based stochastic approximations, the equivalence is lost, which suggests a number of different ways to do regularization. Because this goes beyond what we can analyse theoretically, we extensively study this aspect empirically.

本文探讨强化学习中的KL正则化，发现其内在地平均了q-values，并推导了一个具有两个优点的性能上界。该性能上界在时间序列上的依赖是线性的而非二次的，误差传播方程涉及到估计误差的平均效果而非累积效果。我们也对额外熵正则化的更一般情况进行了研究，这个抽象模型包含了许多现有的强化学习算法。

利用平均值：RL中KL正则化的分析