Large Language Models (LLMs) exhibit powerful summarization abilities.
However, their capabilities on conversational summarization remains under
explored. In this work we evaluate LLMs (approx. 10 billion parameters) on
conversational summarization and showcase their performance on various prompts.
We show that the summaries generated by models depend on the instructions and
the performance of LLMs vary with different instructions sometimes resulting
steep drop in ROUGE scores if prompts are not selected carefully. We also
evaluate the models with human evaluations and discuss the limitations of the
models on conversational summarization

大型语言模型在会话摘要方面的能力有待探索，本研究评估了使用大约 100 亿个参数的语言模型在会话摘要上的性能，展示了其对各种提示的表现，并且证明了模型生成的摘要取决于指令，LLMs 在不同指令下的性能差异，如果提示选择不当，有时会导致 ROUGE 分数的显著下降，还通过人工评估对模型进行了评估，并讨论了模型在会话摘要方面的限制。

零样本对话摘要评估与小型大型语言模型

Zero-shot Conversational Summarization Evaluations with small Large  Language Models

Although pre-trained language models (PLMs) have achieved great success and
become a milestone in NLP, abstractive conversational summarization remains a
challenging but less studied task. The difficulty lies in two aspects. One is
the lack of large-scale conversational summary data. Another is that applying
the existing pre-trained models to this task is tricky because of the
structural dependence within the conversation and its informal expression, etc.
In this work, we first build a large-scale (11M) pretraining dataset called
RCS, based on the multi-person discussions in the Reddit community. We then
present TANet, a thread-aware Transformer-based network. Unlike the existing
pre-trained models that treat a conversation as a sequence of sentences, we
argue that the inherent contextual dependency among the utterances plays an
essential role in understanding the entire conversation and thus propose two
new techniques to incorporate the structural information into our model. The
first is thread-aware attention which is computed by taking into account the
contextual dependency within utterances. Second, we apply thread prediction
loss to predict the relations between utterances. We evaluate our model on four
datasets of real conversations, covering types of meeting transcripts,
customer-service records, and forum threads. Experimental results demonstrate
that TANET achieves a new state-of-the-art in terms of both automatic
evaluation and human judgment.

在 NLP 领域中，尽管预训练语言模型（PLMs）取得了巨大的成功，并成为了一个里程碑，但抽象对话摘要仍然是一个具有挑战性但研究较少的任务。本文中，作者们通过在 Reddit 社区中的多人讨论中构建一个大规模（11M）的预训练数据集 RCS，提出了 TANet，一个基于 Transformer 的网络，借助线程感知的关注机制和线程预测损失函数，有效地解决目前模型的局限性，取得了新的最优效果。