Prioritized Experience Replay (PER) is a technical means of deep
reinforcement learning by selecting experience samples with more knowledge
quantity to improve the training rate of neural network. However, the
non-uniform sampling used in PER inevitably shifts the state-action space
distribution and brings the estimation error of Q-value function. In this
paper, an Attention Loss Adjusted Prioritized (ALAP) Experience Replay
algorithm is proposed, which integrates the improved Self-Attention network
with Double-Sampling mechanism to fit the hyperparameter that can regulate the
importance sampling weights to eliminate the estimation error caused by PER. In
order to verify the effectiveness and generality of the algorithm, the ALAP is
tested with value-function based, policy-gradient based and multi-agent
reinforcement learning algorithms in OPENAI gym, and comparison studies verify
the advantage and efficiency of the proposed training framework.

提出一种 Attention Loss Adjusted Prioritized (ALAP) Experience Replay 算法，结合改进的 Self-Attention 网络和 Double-Sampling 机制，用于拟合超参数以消除 Prioritized Experience Replay (PER) 引起的估计误差。通过与 value-function 基于、policy-gradient 基于以及多智能体强化学习算法在 OPENAI gym 上进行对比研究，验证了该算法的有效性和广泛适用性。