Dialogue is an essential part of human communication and cooperation.
Existing research mainly focuses on short dialogue scenarios in a one-on-one
fashion. However, multi-person interactions in the real world, such as meetings
or interviews, are frequently over a few thousand words. There is still a lack
of corresponding research and powerful tools to understand and process such
long dialogues. Therefore, in this work, we present a pre-training framework
for long dialogue understanding and summarization. Considering the nature of
long conversations, we propose a window-based denoising approach for generative
pre-training. For a dialogue, it corrupts a window of text with
dialogue-inspired noise, and guides the model to reconstruct this window based
on the content of the remaining conversation. Furthermore, to process longer
input, we augment the model with sparse attention which is combined with
conventional attention in a hybrid manner. We conduct extensive experiments on
five datasets of long dialogues, covering tasks of dialogue summarization,
abstractive question answering and topic segmentation. Experimentally, we show
that our pre-trained model DialogLM significantly surpasses the
state-of-the-art models across datasets and tasks. Source code and all the
pre-trained models are available on our GitHub repository
(this https URL).

本文提出了一种用于长对话理解和摘要的预训练框架，其中包括以窗口为基础的去噪方法和稀疏注意力机制，并通过实验表明 DialogLM 预训练模型显著优于现有的模型。