Text classification algorithms investigate the intricate relationships
between words or phrases and attempt to deduce the document's interpretation.
In the last few years, these algorithms have progressed tremendously.
Transformer architecture and sentence encoders have proven to give superior
results on natural language processing tasks. But a major limitation of these
architectures is their applicability for text no longer than a few hundred
words. In this paper, we explore hierarchical transfer learning approaches for
long document classification. We employ pre-trained Universal Sentence Encoder
(USE) and Bidirectional Encoder Representations from Transformers (BERT) in a
hierarchical setup to capture better representations efficiently. Our proposed
models are conceptually simple where we divide the input data into chunks and
then pass this through base models of BERT and USE. Then output representation
for each chunk is then propagated through a shallow neural network comprising
of LSTMs or CNNs for classifying the text data. These extensions are evaluated
on 6 benchmark datasets. We show that USE + CNN/LSTM performs better than its
stand-alone baseline. Whereas the BERT + CNN/LSTM performs on par with its
stand-alone counterpart. However, the hierarchical BERT models are still
desirable as it avoids the quadratic complexity of the attention mechanism in
BERT. Along with the hierarchical approaches, this work also provides a
comparison of different deep learning algorithms like USE, BERT, HAN,
Longformer, and BigBird for long document classification. The Longformer
approach consistently performs well on most of the datasets.

本文研究使用分层传递学习方法进行长文本分类，通过将数据分成块，然后传递到使用双向编码器表示的预训练通用句子编码器和 BERT 的基本模型，然后使每个块的输出表示通过包括 LSTMs 或 CNNs 的浅层神经网络来分类文本数据。使用此扩展方法对 6 个基准数据集进行评估，同时与多个深度学习算法进行比较，其中 Longformer 方法在大多数数据集上始终表现良好。