The synergies between Quality-Diversity (QD) and Deep Reinforcement Learning
(RL) have led to powerful hybrid QD-RL algorithms that have shown tremendous
potential, and brings the best of both fields. However, only a single deep RL
algorithm (TD3) has been used in prior hybrid methods despite notable progress
made by other RL algorithms. Additionally, there are fundamental differences in
the optimization procedures between QD and RL which would benefit from a more
principled approach. We propose Generalized Actor-Critic QD-RL, a unified
modular framework for actor-critic deep RL methods in the QD-RL setting. This
framework provides a path to study insights from Deep RL in the QD-RL setting,
which is an important and efficient way to make progress in QD-RL. We introduce
two new algorithms, PGA-ME (SAC) and PGA-ME (DroQ) which apply recent
advancements in Deep RL to the QD-RL setting, and solves the humanoid
environment which was not possible using existing QD-RL algorithms. However, we
also find that not all insights from Deep RL can be effectively translated to
QD-RL. Critically, this work also demonstrates that the actor-critic models in
QD-RL are generally insufficiently trained and performance gains can be
achieved without any additional environment evaluations.

研究提出了一个统一的模块化框架，名为通用 Actor-Critic Quality-Diversity Deep Reinforcement Learning，以探究深度强化学习中的洞察，并将其应用于 Quality-Diversity 算法，进而提出 PGA-ME (SAC) 和 PGA-ME（DroQ）算法以解决人性化难题，同时演示了境外评估对演员批评模型的必要性。