Consider the problem of training robustly capable agents. One approach is to
generate a diverse collection of agent polices. Training can then be viewed as
a quality diversity (QD) optimization problem, where we search for a collection
of performant policies that are diverse with respect to quantified behavior.
Recent work shows that differentiable quality diversity (DQD) algorithms
greatly accelerate QD optimization when exact gradients are available. However,
agent policies typically assume that the environment is not differentiable. To
apply DQD algorithms to training agent policies, we must approximate gradients
for performance and behavior. We propose two variants of the current
state-of-the-art DQD algorithm that compute gradients via approximation methods
common in reinforcement learning (RL). We evaluate our approach on four
simulated locomotion tasks. One variant achieves results comparable to the
current state-of-the-art in combining QD and RL, while the other performs
comparably in two locomotion tasks. These results provide insight into the
limitations of current DQD algorithms in domains where gradients must be
approximated. Source code is available at this https URL

探讨在训练鲁棒性强的机器人智能体中，使用多样化的代理策略以提高质量多样性优化问题的效率。通过两种导数近似方法实现多样性质量优化算法，并使用四种机器人仿真测试其性能，揭示当前算法在需要近似梯度的领域中的局限性。