Although achieving great success, Large Language Models (LLMs) usually suffer
from unreliable hallucinations. In this paper, we define a new task of
Knowledge-aware Language Model Attribution (KaLMA) that improves upon three
core concerns on conventional attributed LMs. First, we extend attribution
source from unstructured texts to Knowledge Graph (KG), whose rich structures
benefit both the attribution performance and working scenarios. Second, we
propose a new ``Conscious Incompetence" setting considering the incomplete
knowledge repository, where the model identifies the need for supporting
knowledge beyond the provided KG. Third, we propose a comprehensive automatic
evaluation metric encompassing text quality, citation quality, and text
citation alignment. To implement the above innovations, we build a dataset in
biography domain BioKaLMA via a well-designed evolutionary question generation
strategy, to control the question complexity and necessary knowledge to the
answer. For evaluation, we develop a baseline solution and demonstrate the room
for improvement in LLMs' citation generation, emphasizing the importance of
incorporating the "Conscious Incompetence" setting, and the critical role of
retrieval accuracy.

大型语言模型（LLMs）在可靠性方面通常存在不可靠的幻觉。在本文中，我们定义了一项名为知识感知语言模型归因（KaLMA）的新任务，它改进了传统归因语言模型的三个核心问题。

朝可验证的生成方式迈进：知识感知语言模型归因的基准

Towards Verifiable Generation: A Benchmark for Knowledge-aware Language  Model Attribution

Automatic related work generation must ground their outputs to the content of
the cited papers to avoid non-factual hallucinations, but due to the length of
scientific documents, existing abstractive approaches have conditioned only on
the cited paper \textit{abstracts}. We demonstrate that the abstract is not
always the most appropriate input for citation generation and that models
trained in this way learn to hallucinate. We propose to condition instead on
the \textit{cited text span} (CTS) as an alternative to the abstract. Because
manual CTS annotation is extremely time- and labor-intensive, we experiment
with automatic, ROUGE-based labeling of candidate CTS sentences, achieving
sufficiently strong performance to substitute for expensive human annotations,
and we propose a human-in-the-loop, keyword-based CTS retrieval approach that
makes generating citation texts grounded in the full text of cited papers both
promising and practical.

基于引用文献的自动相关工作生成应当将输出结果与引用论文的内容相关联，以避免虚构产生了，然而由于科学文献的篇幅限制，现有的抽象方法只依赖于引用论文的摘要。我们证明了摘要并不总是产生引文时最适合的输入，这样训练出来的模型会出现虚构结果。相反，我们提出使用引用文本片段作为替代输入。由于手动标注引文文本片段非常耗时耗力，我们尝试了基于 ROUGE 的候选引文文本片段标注，并取得了足够强的性能，可以代替昂贵的人工标注，并提出了基于关键词的引文文本片段检索方法，使得生成以引用论文的全文为基础的引文文本成为可能也变得实际可行。

引文文本生成之引文部分识别

Cited Text Spans for Citation Text Generation

The aim of citation generation is usually to automatically generate a
citation sentence that refers to a chosen paper in the context of a manuscript.
However, a rigid citation generation process is at odds with an author's desire
to control the generated text based on certain attributes, such as 1) the
citation intent of e.g. either introducing background information or comparing
results; 2) keywords that should appear in the citation text; or 3) specific
sentences in the cited paper that characterize the citation content. To provide
these degrees of freedom, we present a controllable citation generation system.
In data from a large corpus, we first parse the attributes of each citation
sentence and use these as additional input sources during training of the
BART-based abstractive summarizer. We further develop an attribute suggestion
module that infers the citation intent and suggests relevant keywords and
sentences that users can select to tune the generation. Our framework gives
users more control over generated citations, outperforming citation generation
models without attribute awareness in both ROUGE and human evaluations.

提供了可控的引文生成系统，使用大型语料库中引文句子的属性作为训练的输入源，并开发了属性建议模块来推断引文意图和建议相关关键词和句子，使用户更加掌控生成的引文，与没有属性意识的引文生成模型相比，在 ROUGE 和人类评估中性能更好。