Multi-turn dialogues are characterized by their extended length and the
presence of turn-taking conversations. Traditional language models often
overlook the distinct features of these dialogues by treating them as regular
text. In this paper, we propose a speaker-enhanced pre-training method for long
dialogue summarization, which leverages the inherent structure of multiple-turn
dialogues. To support our study, we curate a diverse dataset that includes
transcripts from real-world scenarios, movie or TV show transcripts, and
dialogues generated by a Large Language Model. We then perform a pre-training,
which encompasses the detection of speaker changes, and masked utterance
generation. Experimental results of fine-tuned models demonstrate that our
model achieves state-of-the-art performance on downstream benchmarks with long
context, surpassing baseline models and highlighting the effectiveness of our
approach. Our findings highlight the importance of curating pre-training
datasets that exhibit diversity and variations in length distribution to ensure
effective alignment with downstream datasets.

在本文中，我们提出了一种用于长对话摘要的说话者增强的预训练方法，该方法利用了多轮对话的内在结构。我们通过收集多样化的数据集，并进行预训练，实验证明我们的模型在具有长上下文的基准测试中取得了最先进的性能，超过了基准模型，并突出了我们方法的有效性。我们的研究结果强调了构建多样化的预训练数据集以确保与下游数据集的有效对应的重要性。