Large language models (LLMs) have achieved remarkable success but still tend
to generate factually erroneous responses, a phenomenon known as hallucination.
A recent trend is to use preference learning to fine-tune models to align with
factuality. However, existing work primarily evaluates fine-tuned models on
in-domain (ID) datasets and the factuality on out-of-domain (OOD) datasets
remains underexplored. In this paper, we conduct a comprehensive evaluation of
the factuality of different models tuned by various preference learning
algorithms and demonstrate that their performance on OOD datasets either
increases minimally or decreases. Subsequently, we reveal that the main cause
of model's failure to uphold factuality under a distribution shift is
\textbf{under-alignment}, rather than \textbf{over-alignment}, by analyzing the
token distribution shift of the models before and after tuning. Finally, we
propose \textbf{APEFT} (\textbf{A}tomic \textbf{P}reference \textbf{E}nhanced
\textbf{F}actuality \textbf{T}uning), a framework that enhances model's
awareness of factuality at the granularity of individual facts. Extensive
experiments demonstrate that APEFT improves model performance by an average of
$\boldsymbol{3.45\%}$ on both ID and OOD datasets, which is highly effective.

该研究针对大型语言模型在事实性上出现的错误回应现象，通过偏好学习方法对模型进行微调以提升事实性，并提出了 APEFT 框架，通过加强模型对个体事实的认识，提高了模型在不同数据集上的表现，平均提升了 3.45%。