Alignment with human preference prevents large language models (LLMs) from
generating misleading or toxic content while requiring high-cost human
feedback. Assuming resources of human annotation are limited, there are two
different ways of allocating considered: more diverse PROMPTS or more diverse
RESPONSES to be labeled. Nonetheless, a straightforward comparison between
their impact is absent. In this work, we first control the diversity of both
sides according to the number of samples for fine-tuning, which can directly
reflect their influence. We find that instead of numerous prompts, more
responses but fewer prompts better trigger LLMs for human alignment.
Additionally, the concept of diversity for prompts can be more complex than
responses that are typically quantified by single digits. Consequently, a new
formulation of prompt diversity is proposed, further implying a linear
correlation with the final performance of LLMs after fine-tuning. We also
leverage it on data augmentation and conduct experiments to show its effect on
different algorithms.

通过控制文本样本数量，本研究发现更多回复但更少提示比大量的提示更能触发语言模型与人类一致，此外，提出了提示多样性的新公式，并使用数据增强进行了实验证明其对不同算法的效果。

人工对齐中的语言模型微调中的数据多样性扩展

Scaling Data Diversity for Fine-Tuning Language Models in Human  Alignment

Large language models (LLMs) are documented to struggle in settings that
require complex reasoning. Nevertheless, instructing the model to break down
the problem into smaller reasoning steps (Wei et al., 2022), or ensembling
various generations through modifying decoding steps (Wang et al., 2023) boosts
performance. Current methods assume that the input prompt is fixed and expect
the decoding strategies to introduce the diversity needed for ensembling. In
this work, we relax this assumption and discuss how one can create and leverage
variations of the input prompt as a means to diversity of thought to improve
model performance. We propose a method that automatically improves prompt
diversity by soliciting feedback from the LLM to ideate approaches that fit for
the problem. We then ensemble the diverse prompts in our method DIV-SE (DIVerse
reasoning path Self-Ensemble) across multiple inference calls. We also propose
a cost-effective alternative where diverse prompts are used within a single
inference call; we call this IDIV-SE (In-call DIVerse reasoning path
Self-Ensemble). Under a fixed generation budget, DIV-SE and IDIV-SE outperform
the previously discussed baselines using both GPT-3.5 and GPT-4 on several
reasoning benchmarks, without modifying the decoding process. Additionally,
DIV-SE advances state-of-the-art performance on recent planning benchmarks
(Valmeekam et al., 2023), exceeding the highest previously reported accuracy by
at least 29.6 percentage points on the most challenging 4/5 Blocksworld task.
Our results shed light on how to enforce prompt diversity toward LLM reasoning
and thereby improve the pareto frontier of the accuracy-cost trade-off.

通过改变输入提示的多样性，并借助多种演绎方法，我们提出的 DIV-SE 和 IDIV-SE 方法在不改变解码过程的前提下，通过多种推理调用和单次推理调用中的多样提示，在固定生成预算的情况下，在多个推理基准和最新的计划基准上，优于现有基线，并在最有挑战的 4/5 Blocksworld 任务上超过先前报告的最高准确率至少 29.6 个百分点，从而改进了 LLM 推理的准确性 - 成本权衡的帕累托前沿。