In this paper, we investigate the use of large language models (LLMs) like
ChatGPT for document-grounded response generation in the context of
information-seeking dialogues. For evaluation, we use the MultiDoc2Dial corpus
of task-oriented dialogues in four social service domains previously used in
the DialDoc 2022 Shared Task. Information-seeking dialogue turns are grounded
in multiple documents providing relevant information. We generate dialogue
completion responses by prompting a ChatGPT model, using two methods:
Chat-Completion and LlamaIndex. ChatCompletion uses knowledge from ChatGPT
model pretraining while LlamaIndex also extracts relevant information from
documents. Observing that document-grounded response generation via LLMs cannot
be adequately assessed by automatic evaluation metrics as they are
significantly more verbose, we perform a human evaluation where annotators rate
the output of the shared task winning system, the two Chat-GPT variants
outputs, and human responses. While both ChatGPT variants are more likely to
include information not present in the relevant segments, possibly including a
presence of hallucinations, they are rated higher than both the shared task
winning system and human responses.

该论文研究了在信息寻求对话中使用大规模语言模型（LLMs）如 ChatGPT 进行基于文件的响应生成。通过人工评估我们发现，ChatGPT 变种虽然有可能插入不在相关片段中的信息，可能存在错觉，但评分比共享任务获胜系统和人工响应更高。

评估信息查询对话中基于文档的响应生成的大型语言模型

Evaluating Large Language Models for Document-grounded Response  Generation in Information-Seeking Dialogues

In this work, we present a model for document-grounded response generation in
dialog that is decomposed into two components according to Bayes theorem. One
component is a traditional ungrounded response generation model and the other
component models the reconstruction of the grounding document based on the
dialog context and generated response. We propose different approximate
decoding schemes and evaluate our approach on multiple open-domain and
task-oriented document-grounded dialog datasets. Our experiments show that the
model is more factual in terms of automatic factuality metrics than the
baseline model. Furthermore, we outline how introducing scaling factors between
the components allows for controlling the tradeoff between factuality and
fluency in the model output. Finally, we compare our approach to a recently
proposed method to control factuality in grounded dialog, CTRL
(arXiv:2107.06963), and show that both approaches can be combined to achieve
additional improvements.

本文提出了一种模型，通过 Bayes 定理将其分解为两个部分，其中一个是传统的非基础回应生成模型，另一个是基于对话上下文和生成的回应重建基础文档的模型。我们提出了不同的近似解码方案，并在多个开放域和面向任务的基础对话数据集上评估我们的方法，实验表明该模型在自动事实度指标方面比基线模型更为准确。此外，我们概述了如何介绍组件之间的缩放因子以控制模型输出中的事实度和流畅性之间的权衡。最后，我们将我们的方法与最近提出的用于控制基础对话中事实度的方法 CTRL 进行了比较，并表明这两种方法可以结合起来实现额外的改进。