Large Language Models (LLMs) often generate erroneous outputs, known as
hallucinations, due to their limitations in discerning questions beyond their
knowledge scope. While addressing hallucination has been a focal point in
research, previous efforts primarily concentrate on enhancing correctness
without giving due consideration to the significance of rejection mechanisms.
In this paper, we conduct a comprehensive examination of the role of rejection,
introducing the notion of model reliability along with corresponding metrics.
These metrics measure the model's ability to provide accurate responses while
adeptly rejecting questions exceeding its knowledge boundaries, thereby
minimizing hallucinations. To improve the inherent reliability of LLMs, we
present a novel alignment framework called Reinforcement Learning from
Knowledge Feedback (RLKF). RLKF leverages knowledge feedback to dynamically
determine the model's knowledge boundary and trains a reliable reward model to
encourage the refusal of out-of-knowledge questions. Experimental results on
mathematical questions affirm the substantial efficacy of RLKF in significantly
enhancing LLM reliability.

通过引入拒绝机制和可靠性度量，本文提出了一种新的对齐框架 RLKF，利用知识反馈动态确定模型的知识边界，并训练可靠的奖励模型来鼓励拒绝超出知识范围的问题，实验证实 RLKF 在显著提高大型语言模型可靠性方面的极大功效。