Quality-Diversity (QD) algorithms can discover large and complex behavioural
repertoires consisting of both diverse and high-performing skills. However, the
generation of behavioural repertoires has mainly been limited to simulation
environments instead of real-world learning. This is because existing QD
algorithms need large numbers of evaluations as well as episodic resets, which
require manual human supervision and interventions. This paper proposes
Reset-Free Quality-Diversity optimization (RF-QD) as a step towards autonomous
learning for robotics in open-ended environments. We build on Dynamics-Aware
Quality-Diversity (DA-QD) and introduce a behaviour selection policy that
leverages the diversity of the imagined repertoire and environmental
information to intelligently select of behaviours that can act as automatic
resets. We demonstrate this through a task of learning to walk within defined
training zones with obstacles. Our experiments show that we can learn full
repertoires of legged locomotion controllers autonomously without manual resets
with high sample efficiency in spite of harsh safety constraints. Finally,
using an ablation of different target objectives, we show that it is important
for RF-QD to have diverse types solutions available for the behaviour selection
policy over solutions optimised with a specific objective. Videos and code
available at this https URL.

本研究提出了 Reset-Free Quality-Diversity optimization (RF-QD) 算法来实现具有多样性和高性能技能的行为副本的自主学习，我们在 Dynamics-Aware Quality-Diversity (DA-QD) 基础上引入了一种行为选择策略，并演示了通过训练自我重置的步态行走任务，可以高效学习到具有多样性和一定安全性的行走技能。