We introduce Language Feedback Models (LFMs) that identify desirable behaviour - actions that help achieve tasks specified in the instruction - for imitation learning in instruction following. To train LFMs, we obtain feedback from Large Language Models (LLMs) on visual trajectories verbalized to language descriptions. First, by using LFMs to identify desirable behaviour to imitate, we improve in task-completion rate over strong behavioural cloning baselines on three distinct language grounding environments (Touchdown, ScienceWorld, and ALFWorld). Second, LFMs outperform using LLMs as experts to directly predict actions, when controlling for the number of LLM output tokens. Third, LFMs generalize to unseen environments, improving task-completion rate by 3.5-12.0% through one round of adaptation. Finally, LFM can be modified to provide human-interpretable feedback without performance loss, allowing human verification of desirable behaviour for imitation learning.

我们引入了语言反馈模型（LFMs），用于在指令跟随的模仿学习中识别理想行为-有助于实现指令中所述任务的行为。通过使用LFMs识别理想行为进行模仿学习，我们改善了在三个不同的语言基础环境（Touchdown、ScienceWorld和ALFWorld）上强大的行为克隆基线的任务完成率。同时，与使用LLMs直接预测动作相比，LFMs在控制LLMs输出令牌数量的情况下取得了更好的效果。LFMs具有泛化到未见环境的能力，通过一轮适应提高了3.5-12.0%的任务完成率。最后，LFM可以进行修改以提供具有人类可解释性的反馈，而不会损失性能，从而允许人类验证模仿学习中的理想行为。

利用语言反馈模型进行政策改进