Large language models have been widely adopted in natural language
processing, yet they face the challenge of generating unreliable content.
Recent works aim to reduce misinformation and hallucinations by resorting to
attribution as a means to provide evidence (i.e., citations). However, current
attribution methods usually focus on the retrieval stage and automatic
evaluation that neglect mirroring the citation mechanisms in human scholarly
writing to bolster credibility. In this paper, we address these challenges by
modelling the attribution task as preference learning and introducing an
Automatic Preference Optimization (APO) framework. First, we create a curated
collection for post-training with 6,330 examples by collecting and filtering
from existing datasets. Second, considering the high cost of labelling
preference data, we further propose an automatic method to synthesize
attribution preference data resulting in 95,263 pairs. Moreover, inspired by
the human citation process, we further propose a progressive preference
optimization method by leveraging fine-grained information. Extensive
experiments on three datasets (i.e., ASQA, StrategyQA, and ELI5) demonstrate
that APO achieves state-of-the-art citation F1 with higher answer quality.

大型语言模型在自然语言处理中被广泛采用，但是它们面临着生成不可靠内容的挑战。最近的研究旨在通过引用作为证据来减少错误信息和产生幻象。然而，当前的引用方法通常集中在检索阶段和自动评估上，忽视了在人类学术写作中增强可信度的引用机制的反映。本文通过将归因任务建模为偏好学习，并引入自动偏好优化（APO）框架来解决这些挑战。我们首先创建了一个经过训练的精心策划的收集 (6330 个例子)，通过收集和过滤现有数据集。其次，考虑到标记偏好数据的高成本，我们进一步提出了一种自动合成归因偏好数据的方法，得到了 95263 对。此外，受人类引文过程的启发，我们还提出了一种利用细粒度信息的渐进式偏好优化方法。在 ASQA、StrategyQA 和 ELI5 三个数据集上进行了大量实验证明，APO 在引文 F1 指标上达到了最先进的水平，并且具有更高的回答质量。