Reinforcement learning (RL) with sparse and deceptive rewards is challenging
because non-zero rewards are rarely obtained. Hence, the gradient calculated by
the agent can be stochastic and without valid information. Recent studies that
utilize memory buffers of previous experiences can lead to a more efficient
learning process. However, existing methods often require these experiences to
be successful and may overly exploit them, which can cause the agent to adopt
suboptimal behaviors. This paper develops an approach that uses diverse past
trajectories for faster and more efficient online RL, even if these
trajectories are suboptimal or not highly rewarded. The proposed algorithm
combines a policy improvement step with an additional exploration step using
offline demonstration data. The main contribution of this paper is that by
regarding diverse past trajectories as guidance, instead of imitating them, our
method directs its policy to follow and expand past trajectories while still
being able to learn without rewards and approach optimality. Furthermore, a
novel diversity measurement is introduced to maintain the team's diversity and
regulate exploration. The proposed algorithm is evaluated on discrete and
continuous control tasks with sparse and deceptive rewards. Compared with the
existing RL methods, the experimental results indicate that our proposed
algorithm is significantly better than the baseline methods regarding diverse
exploration and avoiding local optima.

通过使用多样的过去轨迹作为指导，而不是模仿它们，本文提出了一种方法，使得在线强化学习更快、更高效，即使这些轨迹是次优的或未获得高奖励；此外，引入了一种新的多样性度量来保持团队的多样性和调节探索。在离散和连续控制任务中评估了所提出的算法，与现有的强化学习方法相比，实验结果表明我们提出的算法在多样化探索和避免局部最优方面显著优于基准方法。