Among the many tasks that Large Language Models (LLMs) have revolutionized is text classification. However, existing approaches for applying pretrained LLMs to text classification predominantly rely on using single token outputs from only the last layer of hidden states. As a result, they suffer from limitations in efficiency, task-specificity, and interpretability. In our work, we contribute an approach that uses all internal representations by employing multiple pooling strategies on all activation and hidden states. Our novel lightweight strategy, Sparsify-then-Classify (STC) first sparsifies task-specific features layer-by-layer, then aggregates across layers for text classification. STC can be applied as a seamless plug-and-play module on top of existing LLMs. Our experiments on a comprehensive set of models and datasets demonstrate that STC not only consistently improves the classification performance of pretrained and fine-tuned models, but is also more efficient for both training and inference, and is more intrinsically interpretable.

我们的研究提出了一种使用所有内部表示的方法，通过在所有激活和隐藏状态上采用多种池化策略，首先逐层稀疏化特定于任务的特征，然后在层之间进行聚合，用于文本分类。我们的实验证明，STC不仅在预训练和微调模型上稳定提高了分类性能，而且在训练和推断速度上更加高效，具有更强的内在可解释性。

稀疏化再分类：从大型语言模型的内部神经元到高效的文本分类器