We propose a self-supervised learning method for long text documents based on
contrastive learning. A key to our method is Shuffle and Divide (SaD), a simple
text augmentation algorithm that sets up a pretext task required for
contrastive updates to BERT-based document embedding. SaD splits a document
into two sub-documents containing randomly shuffled words in the entire
documents. The sub-documents are considered positive examples, leaving all
other documents in the corpus as negatives. After SaD, we repeat the
contrastive update and clustering phases until convergence. It is naturally a
time-consuming, cumbersome task to label text documents, and our method can
help alleviate human efforts, which are most expensive resources in AI. We have
empirically evaluated our method by performing unsupervised text classification
on the 20 Newsgroups, Reuters-21578, BBC, and BBCSport datasets. In particular,
our method pushes the current state-of-the-art, SS-SB-MT, on 20 Newsgroups by
20.94% in accuracy. We also achieve the state-of-the-art performance on
Reuters-21578 and exceptionally-high accuracy performances (over 95%) for
unsupervised classification on the BBC and BBCSport datasets.

本文提出了一种基于对比学习的长文本文档自监督学习方法，其中关键是 Shuffle and Divide（SaD），一种简单的文本增强算法，用于对 BERT 文档嵌入进行对比更新所需的预文本任务。通过对 20 Newsgroups、Reuters-21578、BBC 和 BBCSport 数据集进行无监督文本分类的实证评估，我们的方法在准确性方面取得了最高的性能表现。