Diversity plays a significant role in many problems, such as ensemble
learning, reinforcement learning, and combinatorial optimization. How to define
the diversity measure is a longstanding problem. Many methods rely on expert
experience to define a proper behavior space and then obtain the diversity
measure, which is, however, challenging in many scenarios. In this paper, we
propose the problem of learning a behavior space from human feedback and
present a general method called Diversity from Human Feedback (DivHF) to solve
it. DivHF learns a behavior descriptor consistent with human preference by
querying human feedback. The learned behavior descriptor can be combined with
any distance measure to define a diversity measure. We demonstrate the
effectiveness of DivHF by integrating it with the Quality-Diversity
optimization algorithm MAP-Elites and conducting experiments on the QDax suite.
The results show that DivHF learns a behavior space that aligns better with
human requirements compared to direct data-driven approaches and leads to more
diverse solutions under human preference. Our contributions include formulating
the problem, proposing the DivHF method, and demonstrating its effectiveness
through experiments.

通过人类反馈学习行为空间，结合适当的距离度量，提出了一种名为 DivHF 的通用方法，该方法能更好地符合人类要求，并在人类偏好下得到更多样化的解决方案。

通过人类反馈实现多样性

Diversity from Human Feedback

In the context of neuroevolution, Quality-Diversity algorithms have proven
effective in generating repertoires of diverse and efficient policies by
relying on the definition of a behavior space. A natural goal induced by the
creation of such a repertoire is trying to achieve behaviors on demand, which
can be done by running the corresponding policy from the repertoire. However,
in uncertain environments, two problems arise. First, policies can lack
robustness and repeatability, meaning that multiple episodes under slightly
different conditions often result in very different behaviors. Second, due to
the discrete nature of the repertoire, solutions vary discontinuously. Here we
present a new approach to achieve behavior-conditioned trajectory generation
based on two mechanisms: First, MAP-Elites Low-Spread (ME-LS), which constrains
the selection of solutions to those that are the most consistent in the
behavior space. Second, the Quality-Diversity Transformer (QDT), a
Transformer-based model conditioned on continuous behavior descriptors, which
trains on a dataset generated by policies from a ME-LS repertoire and learns to
autoregressively generate sequences of actions that achieve target behaviors.
Results show that ME-LS produces consistent and robust policies, and that its
combination with the QDT yields a single policy capable of achieving diverse
behaviors on demand with high accuracy.

该研究提出了一种新的方法来实现基于行为的轨迹生成，该方法基于 MAP-Elites Low-Spread 和 Quality-Diversity Transformer 两个机制，实现了在不确定环境下生成具有一致性、鲁棒性和可重复性的多样化解决方案，并能够自主地实现目标行为。