Truthfulness is paramount for large language models (LLMs) as they are
increasingly deployed in real-world applications. However, existing LLMs still
struggle with generating truthful answers and content, as evidenced by their
modest performance on benchmarks like TruthfulQA. To address this issue, we
propose GRAdual self-truTHifying (GRATH), a novel post-processing method to
enhance truthfulness of LLMs. GRATH utilizes out-of-domain question prompts to
generate corresponding answers and adaptively optimizes the model via direct
preference optimization (DPO). Note that during this process, GRATH learns
truthfulness in a self-supervised manner without requiring annotated answers.
In particular, GRATH first generates pairwise truthfulness training data by
prompting the LLM itself, with each pair containing a question and its correct
and incorrect answers. The model is then fine-tuned using DPO to learn from the
difference between answer pairs. Subsequently, GRATH iteratively refines the
truthfulness data and optimizes the model, leading to a gradual improvement in
model truthfulness. Empirically, we evaluate GRATH using different 7B-LLMs and
compare with LLMs with similar or even larger sizes on benchmark datasets. Our
results show that GRATH effectively improves LLMs' truthfulness without
compromising other core capabilities. Notably, GRATH achieves state-of-the-art
performance on TruthfulQA, with MC1 accuracy as 54.71% and MC2 accuracy as
69.10%, which even surpass those on larger-scale models, such as
Llama2-Chat-70B, by 23.62% and 24.18%, respectively.

通过利用 GRAdual self-truTHifying (GRATH) 方法，不需要注释答案，仅使用问题提示和优化模型来提高大型语言模型的真实性，结果表明 GRATH 在 TruthfulQA 上取得了最先进的性能。