Because of the inevitable cost and complexity of transformer and pre-trained
models, efficiency concerns are raised for long text classification. Meanwhile,
in the highly sensitive domains, e.g., healthcare and legal long-text mining,
potential model distrust, yet underrated and underexplored, may hatch vital
apprehension. Existing methods generally segment the long text, encode each
piece with the pre-trained model, and use attention or RNNs to obtain long text
representation for classification. In this work, we propose a simple but
effective model, Segment-aWare multIdimensional PErceptron (SWIPE), to replace
attention/RNNs in the above framework. Unlike prior efforts, SWIPE can
effectively learn the label of the entire text with supervised training, while
perceive the labels of the segments and estimate their contributions to the
long-text labeling in an unsupervised manner. As a general classifier, SWIPE
can endorse different encoders, and it outperforms SOTA models in terms of
classification accuracy and model efficiency. It is noteworthy that SWIPE
achieves superior interpretability to transparentize long text classification
results.

该研究提出了一种名为 SWIPE 的模型，它可以有效地学习整个文本的标签，并以一种无监督的方式感知段落的标签并估计它们对文本标签的贡献，从而在长文本分类中实现了高准确度和模型解释性的突破。

高效可解释长文本分类的多维感知器

Multidimensional Perceptron for Efficient and Explainable Long Text  Classification

Position encoding recently has shown effective in the transformer
architecture. It enables valuable supervision for dependency modeling between
elements at different positions of the sequence. In this paper, we first
investigate various methods to integrate positional information into the
learning process of transformer-based language models. Then, we propose a novel
method named Rotary Position Embedding(RoPE) to effectively leverage the
positional information. Specifically, the proposed RoPE encodes the absolute
position with a rotation matrix and meanwhile incorporates the explicit
relative position dependency in self-attention formulation. Notably, RoPE
enables valuable properties, including the flexibility of sequence length,
decaying inter-token dependency with increasing relative distances, and the
capability of equipping the linear self-attention with relative position
encoding. Finally, we evaluate the enhanced transformer with rotary position
embedding, also called RoFormer, on various long text classification benchmark
datasets. Our experiments show that it consistently overcomes its alternatives.
Furthermore, we provide a theoretical analysis to explain some experimental
results. RoFormer is already integrated into Huggingface:
https://huggingface.co/docs/transformers/model_doc/roformer.

本篇论文研究了在语言模型中如何整合位置信息，并提出了一种名为 RoPE 的方法，它可以将位置信息编码为旋转矩阵，并同时将显式的相对位置依赖性结合到自注意力公式中。实验结果表明，RoPE 使 transformer 在处理长文本分类问题时表现出优越的性能。