Reinforcement learning has seen wide success in finetuning large language
models to better align with instructions via human feedback. The so-called
algorithm, Reinforcement Learning with Human Feedback (RLHF) demonstrates
impressive performance on the GPT series models. However, the underlying
Reinforcement Learning (RL) algorithm is complex and requires an additional
training pipeline for reward and value networks. In this paper, we consider an
alternative approach: converting feedback to instruction by relabeling the
original one and training the model for better alignment in a supervised
manner. Such an algorithm doesn't require any additional parameters except for
the original language model and maximally reuses the pretraining pipeline. To
achieve this, we formulate instruction alignment problem for language models as
a goal-reaching problem in decision making. We propose Hindsight Instruction
Relabeling (HIR), a novel algorithm for aligning language models with
instructions. The resulting two-stage algorithm shed light to a family of
reward-free approaches that utilize the hindsightly relabeled instructions
based on feedback. We evaluate the performance of HIR extensively on 12
challenging BigBench reasoning tasks and show that HIR outperforms the baseline
algorithms and is comparable to or even surpasses supervised finetuning.

本文提出一种基于回顾性指令重新标注的新算法 HIR，通过训练模型使其与指令更好地对齐，以解决语言模型中指令对齐的问题，并从 12 个挑战性的 BigBench 推理任务中的表现证明 HIR 优于基线算法，并且即使超过了有监督微调。

回顾智慧让语言模型成为更好的指令跟随者

The Wisdom of Hindsight Makes Language Models Better Instruction Followers

We present a novel method for aligning a sequence of instructions to a video
of someone carrying out a task. In particular, we focus on the cooking domain,
where the instructions correspond to the recipe. Our technique relies on an HMM
to align the recipe steps to the (automatically generated) speech transcript.
We then refine this alignment using a state-of-the-art visual food detector,
based on a deep convolutional neural network. We show that our technique
outperforms simpler techniques based on keyword spotting. It also enables
interesting applications, such as automatically illustrating recipes with
keyframes, and searching within a video for events of interest.

本研究提出了一种利用隐马尔可夫模型与深度卷积神经网络对菜谱中的指示语音逐一进行对齐的方法，在烹饪方面取得了较好的效果，可实现自动加入关键帧及视频事件搜索等功能。