Transformer-based models have revolutionized the performance of a wide range
of language tasks. Intuitively, one might expect text classification, which
does not necessitate as many high-level representations as generative tasks, to
be comprehensively addressed with the powerful representation capabilities of
Transformers. However, in reality, there remains significant potential for
enhancement, particularly in the areas of multi-class and multi-label
classification of lengthy textual documents and other large files. The
performance of Transformer-based models is mainly hindered by a major
limitation: a restricted input length, e.g., 512 tokens for BERT. While an
increase in GPU memory can marginally extend this limit, practical real-world
applications often operate under constrained GPU resources. In this work, we
tackle the input limit problem from the perspective of correlated multiple
instance learning. The proposed approach, LaFiCMIL, serves as a versatile
framework applicable to various large file classification tasks covering
binary, multi-class, and multi-label classification tasks, spanning various
domains including Natural Language Processing, Programming Language Processing,
and Android Analysis. To evaluate its effectiveness, we employ eight benchmark
datasets pertaining to Long Document Classification, Code Defect Detection, and
Android Malware Detection. Leveraging BERT-family models as feature extractors,
our experimental results demonstrate that LaFiCMIL achieves new
state-of-the-art performance across all benchmark datasets. This is largely
attributable to its capability of scaling BERT up to nearly 20K tokens, running
on a single Tesla V-100 GPU with 32G of memory.

利用 Transformer-based 模型作为特征提取器，我们提出的 LaFiCMIL 方法在各种大文件分类任务中，包括二分类、多类别分类和多标签分类任务，跨越了自然语言处理、编程语言处理和 Android 分析等多个领域，在克服输入限制问题方面取得了出色的性能表现，并在八个基准数据集上实验结果表明，相比单个 Tesla V-100 GPU 的 32G 内存限制下，我们能将 BERT 扩展到近 20K 的令牌数量。

LaFiCMIL：从相关多示例学习的视角重新思考大型文件分类

LaFiCMIL: Rethinking Large File Classification from the Perspective of  Correlated Multiple Instance Learning

Multiple studies have shown that Transformers are remarkably robust to
pruning. Contrary to this received wisdom, we demonstrate that pre-trained
Transformer encoders are surprisingly fragile to the removal of a very small
number of features in the layer outputs (<0.0001% of model weights). In case of
BERT and other pre-trained encoder Transformers, the affected component is the
scaling factors and biases in the LayerNorm. The outliers are high-magnitude
normalization parameters that emerge early in pre-training and show up
consistently in the same dimensional position throughout the model. We show
that disabling them significantly degrades both the MLM loss and the downstream
task performance. This effect is observed across several BERT-family models and
other popular pre-trained Transformer architectures, including BART, XLNet and
ELECTRA; we also show a similar effect in GPT-2.

本文发现虽然大多数研究查明 Transformer 是极为健壮的（指对剪枝的容忍度很高），但预先训练好的 Transformer 编码器对于在层输出中删除非常少的要素（模型权重的 < 0.0001%）却异常脆弱。本文发现，对于 BERT 和其他一些预先训练的编码器 Transformer，受影响的组件是 LayerNorm 中的缩放因子和偏差（scaling factors and biases），异常值是在预先训练中早期出现的高幅度归一化参数，并在整个模型中保持相同的维度位置。我们表明，禁用这些异常值显著降低了 MLM 损失和下游任务的性能。这种影响观察到 BART、XLNet、ELECTRA 等几个 BERT 类型的模型和其他流行的预先训练的 Transformer 架构中，也在 GPT-2 中展现了类似效应。