Reinforcement learning has seen wide success in finetuning large language
models to better align with instructions via human feedback. The so-called
algorithm, Reinforcement Learning with Human Feedback (RLHF) demonstrates
impressive performance on the GPT series models. However, the underlying
Reinforcement Learning (RL) algorithm is complex and requires an additional
training pipeline for reward and value networks. In this paper, we consider an
alternative approach: converting feedback to instruction by relabeling the
original one and training the model for better alignment in a supervised
manner. Such an algorithm doesn't require any additional parameters except for
the original language model and maximally reuses the pretraining pipeline. To
achieve this, we formulate instruction alignment problem for language models as
a goal-reaching problem in decision making. We propose Hindsight Instruction
Relabeling (HIR), a novel algorithm for aligning language models with
instructions. The resulting two-stage algorithm shed light to a family of
reward-free approaches that utilize the hindsightly relabeled instructions
based on feedback. We evaluate the performance of HIR extensively on 12
challenging BigBench reasoning tasks and show that HIR outperforms the baseline
algorithms and is comparable to or even surpasses supervised finetuning.

本文提出一种基于回顾性指令重新标注的新算法 HIR，通过训练模型使其与指令更好地对齐，以解决语言模型中指令对齐的问题，并从 12 个挑战性的 BigBench 推理任务中的表现证明 HIR 优于基线算法，并且即使超过了有监督微调。