Although Large Language Models (LLMs) are effective in performing various NLP
tasks, they still struggle to handle tasks that require extensive, real-world
knowledge, especially when dealing with long-tail facts (facts related to
long-tail entities). This limitation highlights the need to supplement LLMs
with non-parametric knowledge. To address this issue, we analysed the effects
of different types of non-parametric knowledge, including textual passage and
knowledge graphs (KGs). Since LLMs have probably seen the majority of factual
question-answering datasets already, to facilitate our analysis, we proposed a
fully automatic pipeline for creating a benchmark that requires knowledge of
long-tail facts for answering the involved questions. Using this pipeline, we
introduce the LTGen benchmark. We evaluate state-of-the-art LLMs in different
knowledge settings using the proposed benchmark. Our experiments show that LLMs
alone struggle with answering these questions, especially when the long-tail
level is high or rich knowledge is required. Nonetheless, the performance of
the same models improved significantly when they were prompted with
non-parametric knowledge. We observed that, in most cases, prompting LLMs with
KG triples surpasses passage-based prompting using a state-of-the-art
retriever. In addition, while prompting LLMs with both KG triples and documents
does not consistently improve knowledge coverage, it can dramatically reduce
hallucinations in the generated content.

大型语言模型在各种自然语言处理任务中表现出色，但在处理需要广泛、现实世界知识的任务，特别是那些涉及长尾实体的任务时，仍然存在困难。为了解决这个问题，本研究分析了不同类型的非参数化知识对语言模型的影响，其中包括文本片段和知识图谱。通过创建一个需要长尾事实知识来回答问题的基准测试工具，我们评估了最先进的语言模型在不同知识环境下的表现。实验结果表明，单独使用语言模型来回答这些问题存在困难，特别是在需要大量长尾知识或丰富知识的情况下。然而，当为语言模型提供非参数化知识时，这些模型的性能显著提高。我们观察到，在大多数情况下，使用知识图谱三元组作为提示的语言模型表现优于使用最先进的检索器的段落提示。此外，虽然同时为语言模型提供知识图谱三元组和文档并不能始终改善知识覆盖率，但可以显著减少生成内容中的幻觉。