Deterministic policy gradient algorithms are foundational for actor-critic methods in controlling continuous systems, yet they often encounter inaccuracies due to their dependence on the derivative of the critic's value estimates with respect to input actions. This reliance requires precise action-value gradient computations, a task that proves challenging under function approximation. We introduce an actor-critic algorithm that bypasses the need for such precision by employing a zeroth-order approximation of the action-value gradient through two-point stochastic gradient estimation within the action space. This approach provably and effectively addresses compatibility issues inherent in deterministic policy gradient schemes. Empirical results further demonstrate that our algorithm not only matches but frequently exceeds the performance of current state-of-the-art methods.

本研究解决了确定性策略梯度算法在控制连续系统时因依赖评论家价值估计的导数而导致的不准确问题。通过在行动空间内采用基于两点随机梯度估计的零阶近似，我们提出了一种新的演员-评论家算法，有效地解决了确定性策略梯度方案中固有的兼容性问题。实证结果表明，该算法的性能不仅匹配，而且在许多情况下超越了当前的最先进方法。

适用于演员-评论家算法的兼容梯度近似