Alignment with human preference is a desired property of large language
models (LLMs). Currently, the main alignment approach is based on reinforcement
learning from human feedback (RLHF). Despite the effectiveness of RLHF, it is
intricate to implement and train, thus recent studies explore how to develop
alternative alignment approaches based on supervised fine-tuning (SFT). A major
limitation of SFT is that it essentially does imitation learning, which cannot
fully understand what are the expected behaviors. To address this issue, we
propose an improved alignment approach named FIGA. Different from prior
methods, we incorporate fine-grained (i.e., token or phrase level) quality
signals that are derived by contrasting good and bad responses. Our approach
has made two major contributions. Firstly, we curate a refined alignment
dataset that pairs initial responses and the corresponding revised ones.
Secondly, we devise a new loss function can leverage fine-grained quality
signals to instruct the learning of LLMs for alignment. Extensive experiments
have demonstrated the effectiveness of our approaches by comparing a number of
competitive baselines.

我们提出了一种名为 FIGA 的改进对齐方法，通过利用细粒度的质量信号，即对比好坏回答的方式，指导大型语言模型的对齐学习。实验证明了我们的方法的有效性。