This paper introduces innovative methods in Reinforcement Learning (RL),
focusing on addressing and exploiting estimation biases in Actor-Critic methods
for continuous control tasks, using Deep Double Q-Learning. We propose two
novel algorithms: Expectile Delayed Deep Deterministic Policy Gradient (ExpD3)
and Bias Exploiting - Twin Delayed Deep Deterministic Policy Gradient (BE-TD3).
ExpD3 aims to reduce overestimation bias with a single $Q$ estimate, offering a
balance between computational efficiency and performance, while BE-TD3 is
designed to dynamically select the most advantageous estimation bias during
training. Our extensive experiments across various continuous control tasks
demonstrate the effectiveness of our approaches. We show that these algorithms
can either match or surpass existing methods like TD3, particularly in
environments where estimation biases significantly impact learning. The results
underline the importance of bias exploitation in improving policy learning in
RL.

本文介绍了创新的强化学习方法，重点是解决和利用演员 - 评论家方法中连续控制任务中的估计偏差，使用深度双 Q 学习。我们提出了两种新算法：期望延迟深度确定性策略梯度（ExpD3）和偏差利用 - 双延迟深度确定性策略梯度（BE-TD3）。ExpD3 旨在通过单个 $Q$ 估计值减少过高估计的偏差，从而在计算效率和性能之间取得平衡，而 BE-TD3 则旨在在训练过程中动态选择最有利的估计偏差。我们在各种连续控制任务上进行了广泛的实验，证明了我们方法的有效性。我们证明这些算法可以在估计偏差显著影响学习的环境中与现有方法（如 TD3）相匹配甚至超越。实验结果强调了利用偏差改进强化学习中的策略学习的重要性。