Aligning large language models (LLMs) with human preferences becomes a key
component to obtaining state-of-the-art performance, but it yields a huge cost
to construct a large human-annotated preference dataset. To tackle this
problem, we propose a new framework that boosts the alignment of LLMs through
Self-generated Preference data (Selfie) using only a very small amount of
human-annotated preference data. Our key idea is leveraging the human prior
knowledge within the small (seed) data and progressively improving the
alignment of LLM, by iteratively generating the responses and learning from
them with the self-annotated preference data. To be specific, we propose to
derive the preference label from the logits of LLM to explicitly extract the
model's inherent preference. Compared to the previous approaches using external
reward models or implicit in-context learning, we observe that the proposed
approach is significantly more effective. In addition, we introduce a
noise-aware preference learning algorithm to mitigate the risk of low quality
within generated preference data. Our experimental results demonstrate that the
proposed framework significantly boosts the alignment of LLMs. For example, we
achieve superior alignment performance on AlpacaEval 2.0 with only 3.3\% of the
ground-truth preference labels in the Ultrafeedback data compared to the cases
using the entire data or state-of-the-art baselines.

通过自动生成的优先数据 (Selfie) 和少量的人工标注优先数据，我们提出了一种新的框架，可以显著增强大型语言模型的对齐性能，进一步提取模型的内在偏好。