This paper investigates the capabilities of Large Language Models (LLMs) in
the context of understanding their own knowledge and measuring their
uncertainty. We argue this is an important feature for mitigating
hallucinations. Specifically, we focus on addressing \textit{known-unknown}
questions, characterized by high uncertainty due to the absence of definitive
answers. To facilitate our study, we collect a dataset with new Known-Unknown
Questions (KUQ) and propose a novel categorization scheme to elucidate the
sources of uncertainty. Subsequently, we assess the LLMs' ability to
differentiate between known and unknown questions and classify them
accordingly. Moreover, we evaluate the quality of their answers in an
Open-Ended QA setting. To quantify the uncertainty expressed in the answers, we
create a semantic evaluation method that measures the model's accuracy in
expressing uncertainty between known vs unknown questions.

本论文探讨了大型语言模型在理解自己的知识和衡量自身不确定性方面的能力，并提供了一个新的已知 - 未知问题数据集以及一个分类方法来解释不确定性的来源。通过评估 LM 在分类已知和未知问题以及在开放式问题回答中的质量方面，量化了 LM 表达答案中的不确定性的方法。