This paper presents AFU, an off-policy deep RL algorithm addressing in a new way the challenging "max-Q problem" in Q-learning for continuous action spaces, with a solution based on regression and conditional gradient scaling. AFU has an actor but its critic updates are entirely independent from it. As a consequence, the actor can be chosen freely. In the initial version, AFU-alpha, we employ the same stochastic actor as in Soft Actor-Critic (SAC), but we then study a simple failure mode of SAC and show how AFU can be modified to make actor updates less likely to become trapped in local optima, resulting in a second version of the algorithm, AFU-beta. Experimental results demonstrate the sample efficiency of both versions of AFU, marking it as the first model-free off-policy algorithm competitive with state-of-the-art actor-critic methods while departing from the actor-critic perspective.

该研究论文介绍了一种名为AFU的离策略深度强化学习算法，它通过使用回归和条件梯度缩放的解决方案，以全新的方式解决了连续动作空间中Q学习中具有挑战性的“最大Q问题”。AFU具有一个演员，但它的评论家更新完全独立于其演员，因此演员可以自由选择。实验结果表明，AFU的两个版本都具有样本效率，使其成为首个与最先进的演员-评论家方法竞争的全模型离策略算法。

AFU：用于连续控制的离策略强化学习中的无演员评论家更新