We explore a strategy to handle controversial topics in LLM-based chatbots
based on Wikipedia's Neutral Point of View (NPOV) principle: acknowledge the
absence of a single true answer and surface multiple perspectives. We frame
this as retrieval augmented generation, where perspectives are retrieved from a
knowledge base and the LLM is tasked with generating a fluent and faithful
response from the given perspectives. As a starting point, we use a
deterministic retrieval system and then focus on common LLM failure modes that
arise during this approach to text generation, namely hallucination and
coverage errors. We propose and evaluate three methods to detect such errors
based on (1) word-overlap, (2) salience, and (3) LLM-based classifiers. Our
results demonstrate that LLM-based classifiers, even when trained only on
synthetic errors, achieve high error detection performance, with ROC AUC scores
of 95.3% for hallucination and 90.5% for coverage error detection on
unambiguous error cases. We show that when no training data is available, our
other methods still yield good results on hallucination (84.0%) and coverage
error (85.2%) detection.

基于维基百科的中立观点原则，我们探索了一种应对 LLM 聊天机器人中有争议话题的策略：承认单一真实答案的缺失，并呈现多个观点。我们将其框架为检索增强生成，其中观点从知识库中检索出来，LLM 的任务是从给定的观点中生成流畅而忠实的回应。我们使用一种确定性的检索系统作为起点，然后关注在这种文本生成方法中出现的常见 LLM 故障模式，即幻觉和覆盖错误。我们提出并评估了三种基于（1）词重叠，（2）重要性和（3）基于 LLM 的分类器来检测这些错误的方法。我们的结果表明，即使只在合成错误上进行训练，基于 LLM 的分类器在错误检测性能上也能达到很高的水平，幻觉检测的 ROC AUC 得分为 95.3％，覆盖错误检测为 90.5％（对于明确的错误案例）。我们展示了当没有可用的训练数据时，我们的其他方法在幻觉检测（84.0％）和覆盖错误检测（85.2％）上仍然产生良好的结果。