Nowadays, pre-trained sequence-to-sequence models such as BERTSUM and BART have shown state-of-the-art results in abstractive summarization. In these models, during fine-tuning, the encoder transforms sentences to context vectors in the latent space and the decoder learns the summary generation task based on the context vectors. In our approach, we consider two clusters of salient and non-salient context vectors, using which the decoder can attend more to salient context vectors for summary generation. For this, we propose a novel clustering transformer layer between the encoder and the decoder, which first generates two clusters of salient and non-salient vectors, and then normalizes and shirinks the clusters to make them apart in the latent space. Our experimental result shows that the proposed model outperforms the existing BART model by learning these distinct cluster patterns, improving up to 4% in ROUGE and 0.3% in BERTScore on average in CNN/DailyMail and XSUM data sets.

我们提出了一种用于摘要生成的新型聚类转换器层，该层在编码器和解码器之间生成两个显着和非显着簇，然后通过对聚类进行归一化和收缩，将它们分隔在潜空间中，从而使解码器可以更多地关注显着的上下文向量，并能够达到比现有 BART 模型更好的性能。

抽象摘要生成中的聚类模式学习