Many recent advances in natural language generation have been fueled by training large language models on internet-scale data. However, this paradigm can lead to models that generate toxic, inaccurate, and unhelpful content, and automatic evaluation metrics often fail to identify these behaviors. As models become more capable, human feedback is an invaluable signal for evaluating and improving models. This survey aims to provide an overview of the recent research that has leveraged human feedback to improve natural language generation. First, we introduce an encompassing formalization of feedback, and identify and organize existing research into a taxonomy following this formalization. Next, we discuss how feedback can be described by its format and objective, and cover the two approaches proposed to use feedback (either for training or decoding): directly using the feedback or training feedback models. We also discuss existing datasets for human-feedback data collection, and concerns surrounding feedback collection. Finally, we provide an overview of the nascent field of AI feedback, which exploits large language models to make judgments based on a set of principles and minimize the need for human intervention.

本文对利用人类反馈来提高自然语言生成的研究进行了综述。通过介绍反馈的形式和目标，讨论了直接使用反馈或训练反馈模型两种方法在训练和解码过程中的应用。此外，我们还探讨了与反馈收集相关的现有数据集和问题，并提供了人工智能反馈领域的概述。

弥合断层：自然语言生成中融入（人类）反馈的调查