Exploration is a key challenge in Reinforcement Learning, especially in
long-horizon, deceptive and sparse-reward environments. For such applications,
population-based approaches have proven effective. Methods such as
Quality-Diversity deals with this by encouraging novel solutions and producing
a diversity of behaviours. However, these methods are driven by either
undirected sampling (i.e. mutations) or use approximated gradients (i.e.
Evolution Strategies) in the parameter space, which makes them highly
sample-inefficient. In this paper, we propose a model-based Quality-Diversity
approach. It extends existing QD methods to use gradients for efficient
exploitation and leverage perturbations in imagination for efficient
exploration. Our approach optimizes all members of a population simultaneously
to maintain both performance and diversity efficiently by leveraging the
effectiveness of QD algorithms as good data generators to train deep models. We
demonstrate that it maintains the divergent search capabilities of
population-based approaches on tasks with deceptive rewards while significantly
improving their sample efficiency and quality of solutions.

本文介绍了一种基于模型的 Quality-Diversity 算法，该算法通过对所有群体成员进行有效的探索和利用想象中的扰动来同时维持性能和高效的多样性，从而提高人群算法在具有欺骗性奖励的任务中的样本效率和解决方案质量。

使用基于模型的品质多样性和梯度的高效探索

Efficient Exploration using Model-Based Quality-Diversity with Gradients

In this work, our goal is to train agents that can coordinate with seen,
unseen as well as human partners in a multi-agent communication environment
involving natural language. Previous work using a single set of agents has
shown great progress in generalizing to known partners, however it struggles
when coordinating with unfamiliar agents. To mitigate that, recent work
explored the use of population-based approaches, where multiple agents interact
with each other with the goal of learning more generic protocols. These
methods, while able to result in good coordination between unseen partners,
still only achieve so in cases of simple languages, thus failing to adapt to
human partners using natural language. We attribute this to the use of static
populations and instead propose a dynamic population-based meta-learning
approach that builds such a population in an iterative manner. We perform a
holistic evaluation of our method on two different referential games, and show
that our agents outperform all prior work when communicating with seen partners
and humans. Furthermore, we analyze the natural language generation skills of
our agents, where we find that our agents also outperform strong baselines.
Finally, we test the robustness of our agents when communicating with
out-of-population agents and carefully test the importance of each component of
our method through ablation studies.

使用动态人口元学习方法来训练代理人在涉及自然语言的多智能体通信环境中与已知、未知和人类伙伴协调。在两个不同的基准博弈中进行了全面评估，并显示出我们的方法在与人类合作时优于所有先前的工作。