Extreme Multi-label Learning (XMC) is a task that allocates the most relevant
labels for an instance from a predefined label set. Extreme Zero-shot XMC
(EZ-XMC) is a special setting of XMC wherein no supervision is provided; only
the instances (raw text of the document) and the predetermined label set are
given. The scenario is designed to address cold-start problems in
categorization and recommendation. Traditional state-of-the-art methods extract
pseudo labels from the document title or segments. These labels from the
document are used to train a zero-shot bi-encoder model. The main issue with
these generated labels is their misalignment with the tagging task. In this
work, we propose a framework to train a small bi-encoder model via the feedback
from the large language model (LLM), the bi-encoder model encodes the document
and labels into embeddings for retrieval. Our approach leverages the zero-shot
ability of LLM to assess the correlation between labels and the document
instead of using the low-quality labels extracted from the document itself. Our
method also guarantees fast inference without the involvement of LLM. The
performance of our approach outperforms the SOTA methods on various datasets
while retaining a similar training time for large datasets.

我们提出了一种通过大型语言模型（LLM）的反馈来训练一个小型双编码器模型的框架，该双编码器模型将文档和标签编码为用于检索的嵌入向量。我们的方法利用 LLM 的零样本能力来评估标签与文档之间的相关性，而不是使用从文档本身提取的低质量标签。我们的方法在各种数据集上表现优于现有技术，同时对于大型数据集具有相似的训练时间，且能保证快速推断。

利用大型语言模型从大规模输出空间中提取间接知识的零 - shot 学习

Zero-Shot Learning Over Large Output Spaces : Utilizing Indirect  Knowledge Extraction from Large Language Models

The eXtreme Multi-label text Classification (XMC) problem concerns finding
most relevant labels for an input text instance from a large label set.
However, the XMC setup faces two challenges: (1) it is not generalizable to
predict unseen labels in dynamic environments, and (2) it requires a large
amount of supervised (instance, label) pairs, which can be difficult to obtain
for emerging domains. Recently, the generalized zero-shot XMC (GZ-XMC) setup
has been studied and ZestXML is proposed accordingly to handle the unseen
labels, which still requires a large number of annotated (instance, label)
pairs. In this paper, we consider a more practical scenario called Extreme
Zero-Shot XMC (EZ-XMC), in which no supervision is needed and merely raw text
of instances and labels are accessible. Few-Shot XMC (FS-XMC), an extension to
EZ-XMC with limited supervision is also investigated. To learn the semantic
embeddings of instances and labels with raw text, we propose to pre-train
Transformer-based encoders with self-supervised contrastive losses.
Specifically, we develop a pre-training method MACLR, which thoroughly
leverages the raw text with techniques including Multi-scale Adaptive
Clustering, Label Regularization, and self-training with pseudo positive pairs.
Experimental results on four public EZ-XMC datasets demonstrate that MACLR
achieves superior performance compared to all other leading baseline methods,
in particular with approximately 5-10% improvement in precision and recall on
average. Moreover, we also show that our pre-trained encoder can be further
improved on FS-XMC when there are a limited number of ground-truth positive
pairs in training. By fine-tuning the encoder on such a few-shot subset, MACLR
still outperforms other extreme classifiers significantly.

本文提出了一种使用预训练的 Transformer 编码器通过自监督对比损失学习实例和标签的语义嵌入的方法，即 MACLR，以解决极端多标签文本分类问题，包括 GZ-XMC、Extreme Zero-Shot XMC 和 Few-Shot XMC，在多个公共 EZ-XMC 数据集上实现了更好的性能。