Deep Reinforcement Learning (RL) has emerged as a powerful paradigm for
training neural policies to solve complex control tasks. However, these
policies tend to be overfit to the exact specifications of the task and
environment they were trained on, and thus do not perform well when conditions
deviate slightly or when composed hierarchically to solve even more complex
tasks. Recent work has shown that training a mixture of policies, as opposed to
a single one, that are driven to explore different regions of the state-action
space can address this shortcoming by generating a diverse set of behaviors,
referred to as skills, that can be collectively used to great effect in
adaptation tasks or for hierarchical planning. This is typically realized by
including a diversity term - often derived from information theory - in the
objective function optimized by RL. However these approaches often require
careful hyperparameter tuning to be effective. In this work, we demonstrate
that less widely-used neuroevolution methods, specifically Quality Diversity
(QD), are a competitive alternative to information-theory-augmented RL for
skill discovery. Through an extensive empirical evaluation comparing eight
state-of-the-art methods on the basis of (i) metrics directly evaluating the
skills' diversity, (ii) the skills' performance on adaptation tasks, and (iii)
the skills' performance when used as primitives for hierarchical planning; QD
methods are found to provide equal, and sometimes improved, performance whilst
being less sensitive to hyperparameters and more scalable. As no single method
is found to provide near-optimal performance across all environments, there is
a rich scope for further research which we support by proposing future
directions and providing optimized open-source implementations.

通过在 8 种最先进的方法上进行广泛的实证评估，本文证明了 Quality Diversity (QD) 方法是技能发现的一种有竞争力的替代方案，既可以提供相同甚至更好的性能，而且对超参数更不敏感且可扩展性更高。