A key aspect of intelligence is the ability to demonstrate a broad spectrum
of behaviors for adapting to unexpected situations. Over the past decade,
advancements in deep reinforcement learning have led to groundbreaking
achievements to solve complex continuous control tasks. However, most
approaches return only one solution specialized for a specific problem. We
introduce Quality-Diversity Actor-Critic (QDAC), an off-policy actor-critic
deep reinforcement learning algorithm that leverages a value function critic
and a successor features critic to learn high-performing and diverse behaviors.
In this framework, the actor optimizes an objective that seamlessly unifies
both critics using constrained optimization to (1) maximize return, while (2)
executing diverse skills. Compared with other Quality-Diversity methods, QDAC
achieves significantly higher performance and more diverse behaviors on six
challenging continuous control locomotion tasks. We also demonstrate that we
can harness the learned skills to adapt better than other baselines to five
perturbed environments. Finally, qualitative analyses showcase a range of
remarkable behaviors, available at: this http URL

智能的一个关键方面是能够展示适应意外情况的广泛行为范围。在过去的十年中，深度强化学习的进展在解决复杂连续控制任务方面取得了突破性成果。本文介绍了一种质量 - 多样性演员 - 评论家（QDAC）的离线演员 - 评论家深度强化学习算法，它利用价值函数评论家和继承特征评论家来学习高性能和多样化的行为。在这个框架中，演员使用受限优化来统一优化两个评论家，以 (1) 最大化回报，同时 (2) 执行多样的技能。与其他质量 - 多样性方法相比，QDAC 在六个具有挑战性的连续控制运动任务上实现了显著更高的性能和更多样化的行为。我们还证明了我们可以利用学到的技能在五个扰动环境中比其他基线更好地进行适应。最后，定性分析展示了一系列非凡的行为。