The development of trustworthy conversational information-seeking systems
relies on dialogue models that can generate faithful and accurate responses
based on relevant knowledge texts. However, two main challenges hinder this
task. Firstly, language models may generate hallucinations due to data biases
present in their pretraining corpus. Secondly, knowledge texts often contain
redundant and irrelevant information that distracts the model's attention from
the relevant text span. Previous works use additional data annotations on the
knowledge texts to learn a knowledge identification module in order to bypass
irrelevant information, but collecting such high-quality span annotations can
be costly. In this work, we leverage reinforcement learning algorithms to
overcome the above challenges by introducing a novel reward function. Our
reward function combines an accuracy metric and a faithfulness metric to
provide a balanced quality judgment of generated responses, which can be used
as a cost-effective approximation to a human preference reward model when only
a few preference annotations are available. Empirical experiments on two
conversational information-seeking datasets demonstrate that our method can
compete with other strong supervised learning baselines.

开发值得信赖的对话式信息搜索系统依赖于能够基于相关知识文本生成忠实准确响应的对话模型。我们通过引入一种新的奖励函数利用强化学习算法来克服数据偏见和冗余信息的两个主要挑战，并在两个对话式信息搜索数据集上的实证实验中展示了我们的方法可以与其他强大的监督学习基准相竞争。