The current winning recipe for automatic summarization is using proprietary
large-scale language models (LLMs) such as ChatGPT as is, or imitation learning
from them as teacher models. While increasingly ubiquitous dependence on such
large-scale language models is convenient, there remains an important question
of whether small-scale models could have achieved competitive results, if we
were to seek an alternative learning method -- that allows for a more
cost-efficient, controllable, yet powerful summarizer. We present InfoSumm, a
novel framework to distill a powerful summarizer based on the
information-theoretic objective for summarization, without relying on either
the LLM's capability or human-written references. To achieve this, we first
propose a novel formulation of the desiderata of summarization (saliency,
faithfulness and brevity) through the lens of mutual information between the
original document and the summary. Based on this formulation, we start off from
Pythia-2.8B as the teacher model, which is not yet capable of summarization,
then self-train the model to optimize for the information-centric measures of
ideal summaries. Distilling from the improved teacher, we arrive at a compact
but powerful summarizer with only 568M parameters that performs competitively
against ChatGPT, without ever relying on ChatGPT's capabilities. Extensive
analysis demonstrates that our approach outperforms in-domain supervised models
in human evaluation, let alone state-of-the-art unsupervised methods, and wins
over ChatGPT in controllable summarization.

InfoSumm 是一种基于信息理论目标的新型框架，通过互信息来蒸馏出一个强大的摘要器，不依赖于大规模语言模型的能力或人工编写的参考文献，参数仅为 568M，性能竞争力强，在人工评估中超过了领域内监督模型，更胜一筹于 ChatGPT 的可控摘要。