Large Language Models (LLMs) have achieved state-of-the-art performance at
zero-shot generation of abstractive summaries for given articles. However,
little is known about the robustness of such a process of zero-shot
summarization. To bridge this gap, we propose relevance paraphrasing, a simple
strategy that can be used to measure the robustness of LLMs as summarizers. The
relevance paraphrasing approach identifies the most relevant sentences that
contribute to generating an ideal summary, and then paraphrases these inputs to
obtain a minimally perturbed dataset. Then, by evaluating model performance for
summarization on both the original and perturbed datasets, we can assess the
LLM's one aspect of robustness. We conduct extensive experiments with relevance
paraphrasing on 4 diverse datasets, as well as 4 LLMs of different sizes
(GPT-3.5-Turbo, Llama-2-13B, Mistral-7B, and Dolly-v2-7B). Our results indicate
that LLMs are not consistent summarizers for the minimally perturbed articles,
necessitating further improvements.

通过评估大型语言模型对原始和扰动数据集上的摘要性能，我们使用关联改写方法来测量大型语言模型作为摘要器的稳健性，并发现大型语言模型对于扰动文章的摘要结果不一致，需要进一步改进。