Factual consistency is an important quality in dialogue summarization. Large
language model (LLM)-based automatic text summarization models generate more
factually consistent summaries compared to those by smaller pretrained language
models, but they face deployment challenges in real-world applications due to
privacy or resource constraints. In this paper, we investigate the use of
symbolic knowledge distillation to improve the factual consistency of smaller
pretrained models for dialogue summarization. We employ zero-shot learning to
extract symbolic knowledge from LLMs, generating both factually consistent
(positive) and inconsistent (negative) summaries. We then apply two contrastive
learning objectives on these summaries to enhance smaller summarization models.
Experiments with BART, PEGASUS, and Flan-T5 indicate that our approach
surpasses strong baselines that rely on complex data augmentation strategies.
Our approach achieves better factual consistency while maintaining coherence,
fluency, and relevance, as confirmed by various automatic evaluation metrics.
We also provide access to the data and code to facilitate future research.

通过使用符号知识蒸馏方法改善小型预训练模型的事实一致性，我们在对话摘要中取得了更好的事实一致性，同时保持了连贯性、流畅性和相关性。

基于大型语言模型的事实对话摘要

Factual Dialogue Summarization via Learning from Large Language Models

The common practice for training commonsense models has gone
from-human-to-corpus-to-machine: humans author commonsense knowledge graphs in
order to train commonsense models. In this work, we investigate an alternative,
from-machine-to-corpus-to-machine: general language models author these
commonsense knowledge graphs to train commonsense models. Our study leads to a
new framework, Symbolic Knowledge Distillation. As with prior art in Knowledge
Distillation (Hinton et al., 2015), our approach uses larger models to teach
smaller models. A key difference is that we distill knowledge symbolically-as
text-in addition to the neural model. We also distill only one aspect-the
commonsense of a general language model teacher, allowing the student to be a
different type, a commonsense model. Altogether, we show that careful prompt
engineering and a separately trained critic model allow us to selectively
distill high-quality causal commonsense from GPT-3, a general language model.
Empirical results demonstrate that, for the first time, a human-authored
commonsense knowledge graph is surpassed by our automatically distilled variant
in all three criteria: quantity, quality, and diversity. In addition, it
results in a neural commonsense model that surpasses the teacher model's
commonsense capabilities despite its 100x smaller size. We apply this to the
ATOMIC resource, and share our new symbolic knowledge graph and commonsense
models.

研究一个新的框架，基于 Symbolic Knowledge Distillation，使用通用语言模型自动化地从 ATOMIC 资源中筛选高质量因果常识，并创造一种新的常识模型，其大小只有原始模型的 1/100，但功能优于其老师模型。