Dual-encoder models have demonstrated significant success in dense retrieval
tasks for open-domain question answering that mostly involves zero-shot and
few-shot scenarios. However, their performance in many-shot retrieval problems
where training data is abundant, such as extreme multi-label classification
(XMC), remains under-explored. Existing empirical evidence suggests that, for
such problems, the dual-encoder method's accuracies lag behind the performance
of state-of-the-art (SOTA) extreme classification methods that grow the number
of learnable parameters linearly with the number of classes. As a result, some
recent extreme classification techniques use a combination of dual-encoders and
a learnable classification head for each class to excel on these tasks. In this
paper, we investigate the potential of "pure" DE models in XMC tasks. Our
findings reveal that when trained correctly standard dual-encoders can match or
outperform SOTA extreme classification methods by up to 2% at Precision@1 even
on the largest XMC datasets while being 20x smaller in terms of the number of
trainable parameters. We further propose a differentiable topk error-based loss
function, which can be used to specifically optimize for Recall@k metrics. We
include our PyTorch implementation along with other resources for reproducing
the results in the supplementary material.

通过研究表明，经过正确训练的标准双编码器模型在极限多标签分类问题中可以匹配或超越状态 - of-the-art 极限分类方法在 Precision@1 方面的性能，并且在可训练参数的数量上小 20 倍。