In many text-generation problems, users may prefer not only a single
response, but a diverse range of high-quality outputs from which to choose.
Quality-diversity (QD) search algorithms aim at such outcomes, by continually
improving and diversifying a population of candidates. However, the
applicability of QD to qualitative domains, like creative writing, has been
limited by the difficulty of algorithmically specifying measures of quality and
diversity. Interestingly, recent developments in language models (LMs) have
enabled guiding search through AI feedback, wherein LMs are prompted in natural
language to evaluate qualitative aspects of text. Leveraging this development,
we introduce Quality-Diversity through AI Feedback (QDAIF), wherein an
evolutionary algorithm applies LMs to both generate variation and evaluate the
quality and diversity of candidate text. When assessed on creative writing
domains, QDAIF covers more of a specified search space with high-quality
samples than do non-QD controls. Further, human evaluation of QDAIF-generated
creative texts validates reasonable agreement between AI and human evaluation.
Our results thus highlight the potential of AI feedback to guide open-ended
search for creative and original solutions, providing a recipe that seemingly
generalizes to many domains and modalities. In this way, QDAIF is a step
towards AI systems that can independently search, diversify, evaluate, and
improve, which are among the core skills underlying human society's capacity
for innovation.

通过 AI 反馈引导的 QDAIF 与非 QD 控制组相比，在创意写作领域覆盖更大范围，生成高质量样本；同时，QDAIF 生成的创意文本经人工评估验证与人类评估具有合理的一致性，突显了 AI 反馈在引导开放式创意搜索方面的潜力。