Large-scale Multi-label Text Classification (LMTC) has a wide range of
Natural Language Processing (NLP) applications and presents interesting
challenges. First, not all labels are well represented in the training set, due
to the very large label set and the skewed label distributions of LMTC
datasets. Also, label hierarchies and differences in human labelling guidelines
may affect graph-aware annotation proximity. Finally, the label hierarchies are
periodically updated, requiring LMTC models capable of zero-shot
generalization. Current state-of-the-art LMTC models employ Label-Wise
Attention Networks (LWANs), which (1) typically treat LMTC as flat multi-label
classification; (2) may use the label hierarchy to improve zero-shot learning,
although this practice is vastly understudied; and (3) have not been combined
with pre-trained Transformers (e.g. BERT), which have led to state-of-the-art
results in several NLP benchmarks. Here, for the first time, we empirically
evaluate a battery of LMTC methods from vanilla LWANs to hierarchical
classification approaches and transfer learning, on frequent, few, and
zero-shot learning on three datasets from different domains. We show that
hierarchical methods based on Probabilistic Label Trees (PLTs) outperform
LWANs. Furthermore, we show that Transformer-based approaches outperform the
state-of-the-art in two of the datasets, and we propose a new state-of-the-art
method which combines BERT with LWANs. Finally, we propose new models that
leverage the label hierarchy to improve few and zero-shot learning, considering
on each dataset a graph-aware annotation proximity measure that we introduce.

本研究在多个数据集上评估了多种 Large-scale Multi-label Text Classification 方法，发现基于概率标签树的层次分类方法优于 Label-Wise Attention Networks。另外，结合 Transformer 的方法在两个数据集上实现了优秀的性能，同时提出了一种新的最先进的方法，将 BERT 与 LWANs 相结合。此外，还引入了图形感知注释接近度度量，提出了利用标签层次结构来改进少量和零 - shot 学习的新模型。

大规模多标签文本分类的实证研究，包括少量和零样本标签

An Empirical Study on Large-Scale Multi-Label Text Classification  Including Few and Zero-Shot Labels

We consider Large-Scale Multi-Label Text Classification (LMTC) in the legal
domain. We release a new dataset of 57k legislative documents from EURLEX,
annotated with ~4.3k EUROVOC labels, which is suitable for LMTC, few- and
zero-shot learning. Experimenting with several neural classifiers, we show that
BIGRUs with label-wise attention perform better than other current state of the
art methods. Domain-specific WORD2VEC and context-sensitive ELMO embeddings
further improve performance. We also find that considering only particular
zones of the documents is sufficient. This allows us to bypass BERT's maximum
text length limit and fine-tune BERT, obtaining the best results in all but
zero-shot learning cases.

该研究在法律领域考虑了大规模多标签文本分类，提出了一个适用于 LMTC、few - 和 zero-shot 学习的新数据集 EURLEX，共包含 57k 个法律文档，注释有～4.3k 个 EUROVOC 标签；实验表明，具有标签注意力的 BIGRU 模型优于当前其他最先进的模型；领域特定的 WORD2VEC 和上下文敏感的 ELMO 嵌入进一步提高了性能；研究人员还发现，只考虑文档中的特定区域就足够了，这使得我们可以规避 BERT 的最大文本长度限制，并使用 fine-tune BERT 在所有情况下获得了最佳结果。