Pretrained Large Language Models (LLMs) have gained significant attention for
addressing open-domain Question Answering (QA). While they exhibit high
accuracy in answering questions related to common knowledge, LLMs encounter
difficulties in learning about uncommon long-tail knowledge (tail entities).
Since manually constructing QA datasets demands substantial human resources,
the types of existing QA datasets are limited, leaving us with a scarcity of
datasets to study the performance of LLMs on tail entities. In this paper, we
propose an automatic approach to generate specialized QA datasets for tail
entities and present the associated research challenges. We conduct extensive
experiments by employing pretrained LLMs on our newly generated long-tail QA
datasets, comparing their performance with and without external resources
including Wikipedia and Wikidata knowledge graphs.

通过预训练大型语言模型解决普适领域问答中的常见知识和罕见长尾知识学习难题，我们提出了一种自动生成专门用于长尾知识的问答数据集的自动方法，并展示了相关的研究挑战。通过预训练语言模型，在我们新生成的长尾问答数据集上进行了详尽实验，比较了其使用维基百科和维基数据知识图等外部资源的性能。

长尾知识的自动问答生成

Automatic Question-Answer Generation for Long-Tail Knowledge

The internet contains a wealth of knowledge -- from the birthdays of
historical figures to tutorials on how to code -- all of which may be learned
by language models. However, there is a huge variability in the number of times
a given piece of information appears on the web. In this paper, we study the
relationship between the knowledge memorized by large language models and the
information in their pre-training datasets. In particular, we show that a
language model's ability to answer a fact-based question relates to how many
documents associated with that question were seen during pre-training. We
identify these relevant documents by entity linking pre-training datasets and
counting documents that contain the same entities as a given question-answer
pair. Our results demonstrate strong correlational and causal relationships
between accuracy and relevant document count for numerous question answering
datasets (e.g., TriviaQA), pre-training corpora (e.g., ROOTS), and model sizes
(e.g., 176B parameters). Moreover, we find that while larger models are better
at learning long-tail knowledge, we estimate that today's models must be scaled
by many orders of magnitude to reach competitive QA performance on questions
with little support in the pre-training data. Finally, we show that
retrieval-augmentation can reduce the dependence on relevant document count,
presenting a promising approach for capturing the long-tail.

本文研究了大型语言模型记忆的知识与其预训练数据中信息之间的关系，结果发现语言模型回答基于事实的问题的能力与预训练期间看到的相关问题文档数量有强相关性和因果关系；同时，该研究发现，虽然更大的模型能学习更长尾的知识，但需要将模型规模增加数个数量级才能在缺乏支持数据的问题上实现具有竞争性的问答表现。最后，对相关文档数量依赖进行检索增强，能够有效捕捉长尾知识。